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@ Improved graphical user interface with method and apparatus for Interfacing to remote devices. 



® An intuitive graphical user interface is based 
upon a geographic map structure, and includes a 
system for controlling remote external electronic de- 
vices. In the defined graphical user interface, each 
Space of the geographic map structure is rendered 
on a touch screen display as a graphic image of a 
geographic space. Within each Space are colored 
cartoon-like icons called "objects- which can be 
Selected and manipulated by the user. Certain ob- 
jects, referred to as portals, transport the user from 
one Space to another Space when Selected. Other 
objects, referred to as buttons, perform associated 
actions or functions when Selected. The graphical 
user interface is displayed on a hand-held display 
device used to control remote devices. Each remote 
electronic device transmits a user interface program 
object that defines a graphical user interface to the 
display device. The hand-hekJ display devices dis- 
plays the graphical user interface associated with the 
renrate device when a user selects an icon asso- 
ciated with the remote device on the display de- 



vice's touch screen display. 
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BACKGROUND OF THE INVENTION 

1. Reld of the invention: 

The present invention relates to the field of 
graphical user Interfaces for computer systems 
and. more particularly, to a method and apparatus 
for providing an intuitive methodology for an oper- 
ator to control remote devices with a computer 
controlled object-oriented user interface utilizing 
animated graphic images. 

2. Art Backgrouncfc 

Most popular graphical user interfaces that are 
available today, such as the Xerox Star®. Apple 
Macintosh®, and Microsoft Windows graphical user 
interfaces, are based upon the "desktop meta- 
phor." In a desktop metaphor system, the display 
screen is treated as a virtual desktop. Graphical 
symbols placed on the virtual desktop <i.e. the 
display screen) are used to represent common 
objects found in an office environment such as 
files, file folders, and printers. These desktop meta- 
phor systems have several limitations which ham- 
per the effectiveness of the man-machine interface 
between the human operator arKJ the computer 
system. 

One problem with the existing desktop meta- 
phor graphical user interfaces is the use of a hier- 
archical file system. Most desktop metaphor 
graphical user interfaces are based upon a com- 
puter having a file system in which the computer 
files are organized in a hierarchical structure. The 
hierarchical file system structure has a root direc- 
tory and several sub-directories within the root di- 
rectory. Each sub-directory n>ay have sub-sub-di- 
rectories, and so on. The hierarchical file system 
structure is graphically illustrated on the screen as 
some type of root file cabinet with file folders 
inside. The Mrcrosoft Windows and Apple Madn- 
loshd operating systems both use hierarchical file 
systems illustrated as fife folders. 

The hierarchical structure of file folders is often 
very confusing to riew users. For example, many 
users get lost while navigating through hierarchical 
file systems because the system of keeping files in 
file folders, whfch are in other file fbkJers nested to 
an art>itrary depth is inherently confusing. Users 
become tost because each file fokfer looks tike 
every other file fokfer, and the user is provided with 
fittle feedback as to his location within the file 
system hierarchy. Also, each file foWer provides no 
information as to what the file folder contains other 
than a user-defined label. For example, except for 
a short text label, which may have been defined by 
another user, the user generally is not atrfe to 
determine whether an open foMer is an accounting 



file fokjer containing spreadsheet data, or a data 
base file folder containing a customer data base. 
Therefore, navigafing through the complex file sys- 
tem hierarchy is difficult since the current desktop 
5 metaphor graphical user interface offers little feed- 
back as to a user's location within the file hierar- 
chy. 

In current graphical user interface systems 
based upon the desktop metaphor, certain applica- 

fo Hons often do not fit well into a desktop metaphor 
environment conceptually. Specifically, non-office 
related applications may be inappropriate for a 
system based upon the desktop metaphor. For 
example, in a desktop metaphor, an application that 

IB controls a television would have to be placed into a 
•paper" file folder on the desktop. 

Another common problem with desktop meta- 
phor user interfaces is that displaying a large num- 
ber of applicatk>ns on the desktop results in a 

20 cluttered desktop. Many users of desktop metaphor 
systems keep large numbers of applications and 
files, usually in icon form, on the display screen. 
Some attempt is made to use a graphical repre- 
sentation that identifies the type of application or 

25 file. However, when a user wishes to open a par- 
ticular appfication or file, the user must search the 
display screen to find the desired fcon just as if the 
user were trying to find a particular paper file on a 
messy desktop. Icons placed the desktop have no 

30 apparent hierarchy, and are all equal. This equality 
may confuse a user, arwJ may create problems 
wfien the user wants to switch from one task to 
another. 

Several systems have evolved to improve the 

35 organization of the desktop metaphor graphical 
user interface. One system is the concept of 
•Roorns". See. Patrick Chan, 'Learning Consid- 
erations in User Interface Design: The Room 
Model'', Report CS-84-16. University of Waterloo 

40 Computer Science Department. Ontario. Canada. 
July. 1984; Brenda Laurel. "Computers as The- 
atre", Addison-Wesley Publishing Co.. 1991; Bren- 
da Laurel. "The Art of Human-Computer Interface 
Design". Addison-Wesley Publishing Co.. 1990; 

45 U.S. Patent 5.072. 412. Henderson. Jr. et al.. 'Wser 
Interface with Multiple Workspaces for Sharing 
Display System Objects^ and the references 
cited therein. 

In systenvs based upon the Rooms concept. 

50 the user creates different rooms where the user 
performs a single task or a group of related tasks. 
Each room represents an individual virtual desktop 
with a group of associated tasks. Each room is 
given a label that kfentifies the room with the 

55 room's functions. For example, a user may create a 
mail room, a typing room with a word processor, 
and an accounting room with a spreadsheet. Each 
room has only those k»ns and windows that are 
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associated with the particular task of the room, as 
well as any windows shared by all rooms, such as 
a cfock. 

To switch between tasks in a system that uses 
Ihe Rooms concept, a user moves between the 
different rooms that are associated with the dif- 
fererit tasks. For example, to go from a spread- 
sheet to retrieve email, a user moves from the 
accounting room to the mail room. To move be- 
tween the two rooms, a user selects an icon that 
represents a door connecting the current room to 
the other room. Users can create doors between 
any of the available rooms. 

Although the Rooms concept helps solve the 
problem of the "cluttered desktop." the Rooms 
concept still relies upon the common desktop met- 
aphor. For example, the Rooms interface concept 
does not help simplify the confusing hierarchical 
file structure used to store information. The Rooms 
concept also does little to solve the form and 
funcfion inconsistencies that are apparent when 
applications which do not fit into the desktop meta- 
phor are added to the system. 

In prior art graphical user interfaces, each time 
a new piece of hardware is introduced to the sys- 
tem a new piece of control software must also be 
added by the user. For example, if a user of an 
Apple® Macintosh® or a user of Microsoft® Win- 
dows desires to add a new printer to the system, 
the user must also install a new printer driver to 
control the added printer. Therefore, each time a 
user desires to add a new device to the system, 
the user must also follow a complicated procedure 
to add the appropriate software. 

As will be described, the present invention 
provides an improved graphical user interface that 
overcomes several of the problems associated with 
desktop metaphor systems. A method and appara- 
tus for obtaining the user interface of a remote 
device without requiring the user to load the related 
software is also disck3sed. The remote device can 
be controlled using the remote device's user inter- 
face as displayed on the present invention's dis- 
play screen. Although the user interface is de- 
scribed with reference to the control of remote 
devices, it will be appreciated from the discussion 
that foltows that the user interface may be applied 
to a v^iety of applications in computer worksta- 
tions, portable computers, and hand held control 
systems. 

SUIWRflARY OF THE INVENTIOW 

The present invention provides an intuitive 
graphical user interface and a method and appara- 
tus for controlling remote devices. The graphical 
user interface of the present Invention is displayed 
on a hand-heW display device. The hand-heW dis- 



play device comprises at least one central process- 
ing unit (CPU) coupled to a graphic display sys- 
tem. In the present embodiment, the graphic dis- 
play system is touch sensitive lo permit the user to 
5 input information using a fingertip without the need 
for additional input devices such as a keytx)ard. A 
communications circuit is also coupled to the CPU 
transmitting information over some communication 
medium. The communications circuit is used to 
to control and lo communicate with remote devices. 
The graphical user interface of the present 
invention creates a worW of Spaces organized into 
• a geographic map structure that is simple to navi- 
gate through since each Space provides familiar 
75 geographic sunroundings. Each Space is represent- 
ed by a background image which provides naviga- 
tional landmarks that identify where the Space is 
located within the geographic map staicture. For 
example, the background image of a living room 
20 Space contains items typically found in a living 
room such as a lamp, a chair, and a tabte. There- 
fore, navigation through this "world" is made easier 
because the user can orient himself by looking 
around and examining the surroundings just as a 
25 person woukJ do in the real worW. 

Within each Space, the user interface provides 
graphic objects that can be selected and manipu- 
lated by the user. The graphic objects are rendered 
as animated cartoon-like graphic objects that are 
30 easily identifiable as real worW objects such as 
televisions and VCRs. There are at least three 
types of objects that exist in the Spaces of the user 
interface: data objects, buttons, and portals. Data 
objects are objects that are associated with a par- 
35 ticular piece of data. Buttons are objects that are 
assodated with a particular function or action; when 
a user selects a button, the user interface performs 
the function or action associated with the button. 
Portals are ot>jects that are displayed in a first 
40 Space but are associated with a second Space; 
when a user selects a portal the user passes 
through the portal and moves to the second Space 
associated with that portal. The user environment is 
further enhanced by the use of sound, coordinated 
45 with every user action and supplementing the vi- 
sual feedback provided. 

To control real world remote devices, many of 
the graphic objects presented on the screen are 
associated with remote devices in the real world. 
50 To control a remote device, a user selects the 
graphic object pk:tured on the display screen asso- 
ciated with the particular remote device. After the 
user selects a graphic object associated with a 
remote device, the hand-held display device dis- 
ss plays a user interface for controlling the remote 
device. The user can then interact with the remote 
device's user interface. To control the remote de- 
vice, the present invention invokes a devk:e <in^T 
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method for the remote device. The device driver 
method may.be located within the remote device or 
within the local hand-held display device, depend- 
ing on the remote device's sophistication. 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 

The objects, features and advantages of the 
present invention will be apparent to one skilled in 
the art. in view of the following detailed description to 
in which: 

FIGURE la illustrates an overview of the remote 
control functionality provided by the present in- 
vention. 

FIGURE lb illustrates an overview of the hard- ts 
ware used to implement the hand-held display 
device of the present invention. 
FIGURE 1c illustrates a block diagram of an 
intelligent remote device that is controlled by 
the present invention. 20 
RGURE Id illustrates a block diagram of a 
simple remote device that is controlled by the 
present invention. 

FIGURE 1e illustrates a block diagram of con- 
ventional electronic device that is controlled re- 25 
motely by the present invention. 
FIGURE 2a illustrates a structure of different 
geographic spaces. 

RGURES 2b - 2c illustrate a sample geographic 
map structure of spaces used to provide a user 30 
interface. 

FIGURE 3 illustrates a view of the outer-pace 
Space of the user interface based upon the 
geographic map structure in Figures 2b and 2c. 
RGURE 4 illustrates a first view of the Earth's as 
surface Space in the user interlace based upon 
the geographk: map structure in Figures 2b and 
2c. 

RGURE 5 illustrates a second view of the 
Earth's surface Space in the user interface 40 
based upon the geographic map structure in • 
Rgures 2b and 2c. 

RGURE 6 illustrates a view of the first house 
Space of the user interface leased upon the 
geographic map structure in Figures 2b and 2c. 45 
FIGURE 7 illustrates a view of a Bving room 
Space in the user interlace based upon the 
geographic map structure in Figures 2b and 2c. 
FIGURE 8 illustrates a view of a television Space 
in the user interface based upon the geographic so 
map structure in Figures 2b and 2c. 
RGURE 9 illustrates a view of a television pro- 
gram fisting book Space in the user interface 
iDased upon the geographic map structure in 
Figures 2b and 2c. 55 
RGURE 10 iBustrates a view of a single televi- 
sion program description Space of the user in- 
terface based upon the geographic map struc- 



ture in Rgures 2b and 2c. 

RGURE 11 illustrates a conceptual view of the 
television program listing book Space of Figure 
9. 

RGURE 12 illustrates a view of a plurality of 
objects which do not fit on a single screen 
display. 

FIGURE 13 illustrates a conceptual view of the 
plurality of objects placed on an object wheel. 
RGURE 14 illustrates a conceptual view of the 
object wheel of Figure 13 where the object 
wheel has rotated revealing new objects. 
RGURE 15 illustrates the Spin gesture being 
used to spin an object wheel. 
FIGURE 16a illustrates a Pickup gesture being 
used to pickup an object off an object wheel. 
FIGURE 16b illustrates a Pickup gesture being 
used to pickup an object from a Space. 
FIGURE 17a illustrates the Agent Portal of the 
present invention wearing a doctor's uniform to 
symbolize a diagnostic mode. 
RGURE 17b illustrates the Agent Portal of the 
present invention holding a pencil to symbolize 
a writing mode. 

FIGURE 17c illustrates the Agent Portal of the 
present invention attempting to alert the user 
about a recent event. 

FIGURE 18a illustrates an intelligent device 
transmitting its graphical user interface to a dis- 
play device. 

FIGURE 18b illustrates a display device display- 
ing the graphical user interface of an intelligent 
controllable device, 

RGURE 18c illustrates the display device trans- 
mitting user interactions with the user interface 
to the intelligent device. 

FIGURE 18d illustrates the intelligent device 
transmitting return values or feedback to the 

display device. 

RGURE 19a illustrates a conceptual diagram of 
the program objects in the intelligent device 
being sent to the display device. 
RGHJRE 19b illustrates a conceptual diagram of 
the program objects of the intelligent device in 
the display device. 

RGURE 19c illustrates a conceptual diagram of 
the display device invoking a method in an 
object within the remote intelligent device. 
RGURE 19d illustrates a conceptual diagram of 
the method in an object within the remote intel- 
ligent device returning status. 
FIGURE 20 illustrates a ftowchart of the transfer 
of an intelligent device's user interface. 
FIGURE 21a illustrates a state diagram of the 
RPC manager for a client requesting a remote 
method invocation. 

RGURE 21b illustrates a state diagram of the 
RPC manager for a server handling a remote 
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method invocation. 

FIGURE 21c illustrates the sequence of a re- 
mote method invocation. 
FIGURE 22a illustrates Ihe simple device trans- 
ferring its graphtcat user interface to a display 
device. 

FIGUFtE 22b illustrates a display device display- 
ing the graphical user interface of the simple 
controllable device. 

RGURE 22c illustrates the display device trans- 
mitting simple control codes to the simple de- 
vice. 

FIGURE 22d illustrates the simple device trans- 
mitting return values or feedback to the (fisplay 
device. 

FIGURE 23a illustrates a conceptual diagram of 
the program objects in a simple device being 
sent to the display device. 
FIGURE 23b illustrates a conceptuaJ diagram of 
the program objects of the simple device in the 
display device. 

RGURE 23c illustrates a conceptual diagram of 
the display device invoking a method in a pro- 
gram object and transmitting control codes to 
the simple device. 

FIGURE 23d illustrates a conceptual diagram of 
the remote simple device returning status in- 
formation. 

RGURE 24 illustrates a flowchart of the transfer 
of a simple device's user interface. 

NOTATION AND NOMENCLATURE 

The detailed descriptions whfch follow are pre- 
sented largely in terms of display images, algo- 
rithms, and symbofic representations of operations 
of data bits within a computer memory. These 
algorithmic descriptions and representations are 
the means used by those skilled in the data pro- 
cessing arts to most effectively convey the sub- 
stance of their work to others skilled in the art. 

An algorithm is here, and generally, conceived 
to iDe a consistent sequence of steps leading to a 
desired result. These steps are those requiring 
physical manipulations of physical quantities. Usu- 
ally, though not necessarily, these quantities take 
the form of electrical or nrtagnetic signals capable 
of being stored, transferred, combined, compared, 
selected, chosen, modified, and otherwise manipu- 
lated. It proves convenient at times, principally for 
reasons of common usage, to refer to these signals 
as bits, values, elements, symbols, characters, im- 
ages, terms, numbers, or the like. It shoukJ be 
borne in mind, however, that all of these, as well as 
similar terms, are to be associated with the appro- 
priate physfcal quantities and are merely conve- 
nient labels applied to these quantities. 



In the present case, the operations are ma- 
chine operations performed in coniunctk>n with a 
human operator. Useful machines for performing 
the operations of the present invention include gen- 
5 oral purpose digital computers or other similar de- 
vices. In all cases, the distinction between the 
method operations of operating a computer and the 
method of computation itself should be kepi in 
mind. The present invention relates to method 
10 steps for operating a computer and processing 
electrical or other physical signals to generate oth- 
er desired physical signals. 

The present invention also relates to apparatus 
for performing these operations. This apparatus 
IS may be specially constructed for the required pur- 
poses or it may comprise a general purpose com- 
puter selectively activated or reconfigured by a 
computer program stored in the computer. The 
algorithms presented herein are not inherently re- 
20 lated to any particular computer or other apparatus. 
In particular, various general purpose machines 
may be used with programs in accordance with the 
teachings herein, or it may prove more convenient 
to construct more specialized apparatus to perform 
25 the reqmed method steps. The required structure 
for a variety of these machines will appear from the 
description given below, fvlachines which may per- 
form the functions of the present invention include 
those manufactured by the Assignee, RrstPerson, 
30 Inc., as well as other manufacturers of computer 
systems, including Sun Microsystems, Inc. of 
Mountain View, California. 

CODING DETAILS 

35 

No particular programming language has fc>een 
indicated for carrying out the various procedures 
described herein. This is due in part to the fact that 
not all languages that might be mentioned are 

40 universally available. Each user of a particular com- 
puter will be aware of a language which is most 
suitable for his immediate purposes, in practice, it 
has proven useful to substantially Implement the 
present invention in a high level language which Is 

45 then compiled into machine executable object 
code. Because the computers and the monitor sys- 
tems which may be used in practicing the instant 
invention consist of many diverse elements, no 
detailed program Bsting has been provided. It is 

50 considered that the operations and other proce- 
dures described herein and illustrated in the ac- 
companying drawings are sufficiently disclosed lo 
permit one of orcfinary skill to practice the instant 
invention. 

55 
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DETAILED DESCRIPTION OF THE INVENTtON 

A computer based remote control system with 
an intuitive graphical user interface is disclosed. In 
the followifig description, for. purposes of explana* 
tion, specific nomenclature is set forth to provide a 
thorough understanding of the present invention. 
However, it wiH be apparent to one skilled in the art 
that these specific details are not required to prac- 
tice the present invention. In other instances, well 
known circuits, functior^s, processes, and devices 
are shown in block diagram and conceptual dia- 
gram form in order not to obscure the present 
invention unnecessarily. 

System Ovefview 

Figure la ilhistrates an overview of the com- 
puter based remote control system of the present 
invention. The main component of the present in- 
vention consists of a hand-held display device 170 
as illustrated in Figure la. In the present embodi- 
ment, the hand-held display device 170 comprises 
a computer system with a color graphic touch 
screen display. The hand-held display device 170 
is designed to control any compatible remote de- 
vice such as thermostat 150. video cassette re- 
corder 39, and stereo system 152. The hand-held 
display device 170 communicates with the remote 
devices using a communications medium 160. 

The remote control system of the present in- 
vention as illustrated in Figure la differs signifi- 
cantly from prior art remote control systems since 
it is based upon modular obiect-oriented software. 
The object-oriented software provides features pre- 
viously not available in prior art remote control 
systems. For example, when an operator (some- 
time referred to herein as a "user") adds a new 
compatible remote device, such as thermostat 150 
to the system, the new remote device transmits 
Information about itself to the hand-held display 
device 170. The transmitted information includes a 
user interface object that can be used to render a 
graphical user interface for the remote device. The 
present invention uses the user interface object to 
display a graphical user interface for the new re- 
mote device on the hand-held display device 170. 
The transmitted information may also incfejde a 
device driver ot>}ect that is used to control the new 
remote device. To organize all the available user 
interface objects, the hand-held display device 170 
aeales a "world " of geographic Spaces that can 
be easily be navigated usirig a set of gestures. 

This Specification wilt first disclose the hard- 
ware required to construct the hand-held display 
device 170 and the compatible remote devices of 
the present invention. The graphicat user interface 
comprising geographic Spaces and objects which 



is displayed on the hand-hekJ display device 170 
will then t>e described. A set of gestures used by 
operators to navigate around the user interface of 
the present invention will also be defined. To com- 

5 bine the user interface elements, a tour through an 
embodiment of the user interface of the present 
invention will be provided to illustrate how the 
navigational gestures are used to interact with the 
objects displayed on the hand-held display device 

JO 170. Rnally. the system of obtaining information 
from the remote devices is defined, such that the 
hand*held display device 170 may communicate 
with and control remote external devices. 

Due to the limitations of a written Specification 

15 to describe animated events on an interface, the 
reader is referred to a video tape entitled "Green 
Project User Interface." submitted by the applicant 
concurrently with the filing of the application on 
which this patent is based. In addition it must also 

20 be noted that the features of the present invention 
are illustrated in black and white within the accom- 
panying figures. However, in the presently pre- 
ferred emtxxjiment. and as descrit>ed below, ob- 
jects and features are displayed in cotor. 

25 

Hardware Configuration 

The hand-held display device 170 of the 
present invention comprises a computer system. 
30 such as that illustrated in Figure lb. The computer 
system 170 of Figure lb includes a central pro- 
cessing unit (CPU) 30. a memory unit 33, and an 
input/output (I/O) unit 34. These elements are those 
typically found in most general purpose computers. 
35 and in fact, the hand-hekl computer system 170 is 
intended to be representative of a broad category 
of data processing devices capable of generating 
graphic displays. 

Coupled to the input/output (I/O) unit 34 of 
40 computer system 170 is a color graphic saeen 
display 37. The screen display 37 comprises, in 
part, a color raster display device such as a cath- 
ode ray tube (CRT) monitor or a liquid crystal 
display (LCD). The screen display 37 must be of 
45 sufTicient resolution such that screen display 37 
can render graphk: images. 

In the preferred embodiment, the screen dis- 
play 37 further comprises a touch screen display 
system. The touch saeen display includes a lea- 
se ture that can detect the preserK:e of a finger touch- 
ing the display screen 37. Furthermore, the touch 
screen can identify the precise k)cation of the 
finger touching the display saeen 37 such that the 
display saeen can sense finger gestures made by 
55 a user's finger on the display saeen 37. The touch 
saeen display 37 may comprise one of a variety of 
touch sensitive display screens commercially avail- 
able on the market 
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Atternatively. the hand-held computer system 
170 may be constructed with a conventional graph- 
ic display system without a touch sensitive display 
screen. Such a system requires an aHernative input 
system to obtain the user's interactions. Figure lb 
illustrates an alternative input device 42. Although 
the user interface of the present invention may be 
used with alternative input devices 42 such as 
mice, trackballs, and joysticks, the system will be 
described within this specification as the preferred 
embodiment with a touch screen display 37. 

In operation, the CPU 30 generates and dis- 
plays a graphical user interface on touch screen 
display 37 through the execution of computer code 
stored in memory 33. The computer code stored in 
memory 33 is not disclosed herein since the de- 
scription of the present invention in this Specifica- 
tion is sufficient for one skilled in the computer art 
to utilize the teachings of the invention in a variety 
of computer systems using one of many computer 
languages. 

Coupled to the input/output (I/O) unit 34 of 
computer system 170, in Figure lb, is an external 
communication circuit 35. The external commu- 
nication circuit 35 provides a system for transmrt- 
ling and receiving information to and from the 
hand-held computer system 170. The present in- 
vention uses the external communication circuit 35 
in computer system 170 to communicate with re- 
mote devices such as a video cassette recorder 39 
and a thermostat 150. across communicatioris me- 
dium 160. The communtcatipn circuit 35 may use 
one of many well-known cqmmunication technol- 
ogies such as radio waves, tow power, cellular, 
infrared signals, or data signals carried on existing 
power lines. 

To be controlled by the present hand-held dis- 
play device 170 of the present invention, a remote 
device must have at least one method for receiving 
information transmitted by the harKi-held display 
device 170. Ideally, the remote devices are intel- 
ligent devices that can both transmit and receive 
information. However, the hand-held display device 
170 can control any remote device that fits within 
any of the three foltowing remote device "types". 

The first type of remote device that can be 
controlled by the present invention is an "intelligent 
remote device" specifically designed to operate 
with the hand-heW display device 170' In the 
present invention, an "intelligent remote devfce" is 
defined as a remote device that contains a micro- 
processor, a communication system, and a mem- 
ory. Figure 1c provides a btock diagram of the 
components in an intelligent renrwte device 150. As 
illustrated in Figure 1c. the intelligent remote de- 
vice 150 comprises a CPU 403. a memory 404. a 
communications circuit 401. and a device control 
circuitry 405. 



The hand-held display device 170 of the 
present invention communicates with an intelligent 
remote device 150 through the intelligent remote 
device's communications circuit 401. The memory 
5 404 stores programs and data used by the CPU 
403. including a user interface object 303 and a 
device driver object 301. The CPU 403 in the 
intelligent remote device invokes methods in the 
device driver object 301 to control the remote 
10 device. 

The second type of remote device that can be 
controlled by the present invention is a "simple 
remote device" specifically designed to operate 
with the hand-held display device 170. In the 
IS present invention, a remote device is defined as a 
"simple remote device" if the remote device does 
not have a microprocessor or the remote device 
contains such a primitive processor that the hand- 
held display device 170 can not invoke methods in 
«i a device driver object within the remote device. 

Figure Id provides a block diagram of the 
components in a typical simple remote device 155. 
As illustrated In Figure Id. the simple remote 
device li55 comprises an emitter 420, a receiver 
25 421, a memory 424. and device control circuitry 
425. The emitter 420 in the simple renwte device 
155 is used to transmit the contents of the memory 
424 to the hand-held display device 170. There- 
fore, the simple remote device 155 transmits both 
30 the user interface object 353 and the device driver 
object 351 in the memory 424, such that the hand- 
hekJ display device receives both objects. 

To control the simple remote device 155. the 
display device 170 invokes a method within the 
35 device driver object 351 to translate, the user's 
interactions into simple predefined control codes. 
The display device 170 then transmits the simple 
predefined control codes to the simple remote de- 
vice 155 which receives the control codes though 
40 receiver 421 and passes the information to the 
device control circuitry 425. 

Finally, the third type of remote device that can 
be controlled by the present inventton is a conven- 
tional electronic device that can be controlled by 
<s transmitted sigr^ls such as infrared signals. There- 
fore, the present invention is backvrard compatible 
vrifh most existing devices that can be controlled 
remotely. Figure 1e illustrates a typical electronic 
device 157 that can be controlled by coded sig- 
50 nab. The device has a receiver 450 lor receiving 
transmitted coded corUrol signals and device con- 
trol circuitry 455 for responding to the coded con- 
trol signals. The hand-heW display device 170 con- 
trols the electronic device 157. by transmitting cod- 
55 ed control signals. The remote electronic device 
157 receives the transmitted signals and resportds 
accordingly. 
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The Display Oevlce User Interface 

(n order to create a new product that is easy to 
use. product designers attempt to make the new 
product operate in a manner similar to an old 
product that ihe Intended operator already knows 
how to operate. For example, audio cassette tape 
recorders have been available lor such a long time 
that most people know how to operate an audio 
cassette tape recorder. The play, stop, rewind, fast- 
forward, and record buttons on cassette tape re- 
corders are therefore familiar to most people. Video 
cassette recorder manufacturers produced easy-to- 
use video cassette recorders by taking advantage 
of this common knowledge. The video cassette 
recorders are easy to use since the video cassette 
recorders used the same buttons, i.e.. the play, 
stop, rewind, last-forward, and record buttons, that 
audio cassette recorders utilize for operation. 
. To make the hand-hekf display device 170 
easy to use. the present inventkMi displays an 
intuitive graphical user interface on the hand-held 
display device 170. This section defines the graphi- 
cal user interface which is provided to users on the 
hand-held display device 170. 

To design a graphical user interface that is 
easy to navigate, designers should organize the 
graphical user interface with a structure that most 
users already understand. Hurnans live in a three 
dimenstonaf geographic worM. To survive in our 
three dimensional geographic world people must 
learn how to navigate around the kx:al geographic 
area. Including such geographic spaces their 
house, their neighborhood, and their town. In order 
to do this, each person creates a mental map of 
geographic spaces of their relevant local world. 

The mental maps of geographic Spaces people 
create are de-facto • geographic map structures. 
Figure 2a illustrates a typical geographic map 
structure. For example, every person on this planet 
knows that they live on the Earth. Each person also 
lives in a certain town: yet, each person Is aware 
that other towns exist on the Earth. Each person 
fives at a particular house on a particular street; 
yet, each person is aware that there are other 
streets in the town with other houses on each 
street. The identification of one partkujiar house on 
one street within one town on Earth is a naturally 
occurring geographk: map structure as illustrated in 
Figure 2a. 

The geogr^k: map structure can be ex- 
tended into the house. For example, the living 
room is one room of many within the house. The 
television, television program schedule and the vid- 
eo cassette recorder are objects within the living 
room. Each object can have its own geographic 
space. The map sUucture can also be extended to 
abstract spaces as well as geographk: spaces. 



Each television program listed in a televiskm pro- 
gram schedule has a program description. The 
program description can be considered a space 
within the television program schedule. Most peo- 

5 pie understand both the structure of geographic 
spaces and the detail that wilt be fourxi within each 
geographic space. Therefore the de-facto geo- 
graphic map structure of spaces represents a 
structure that is familiar to most people. 

10 The present inventk>n uses the familiar geo- 
graphic map structure of our three dimensional 
worW in a graphical user interface. The present 
invention describes a technique of user navigatbn 
In an arlDitrarity large and complex geographic map 

15 structure through graphic background images that 
depict geographic spaces. Such geographic nr«p 
structures improve a user's memory of complex 
data organization. The advantage of including such 
geographic map structures is that the familiar struc- 

20 tures enable a wider range of people to use in- 
formation storage facilitated by a computer. 

Figures 2b and 2c illustrate a sample geo- 
graphic map structure utilized by the present in- 
vention. In the present invention, each Space in the 

25 geographic map structures of Figures 2b and 2c 
is riepresented as a background image on the dis- 
play screen of hand-held display device such as 
the one illustrated in Figure lb. Located within 
each geographic Space are objects that can be 

30 selected arxJ manipulated by the user. 

User Interface Display Elements 

The user interface environment of the present 

35 invention consists of two main display elements: 
"Spaces" and selectable "objects." Each area of 
the geographk: map structure of Figures 2b and 
2c represents a Space. The Space provides a 
geographk: reference that Informs a user of the 

40 user's current tocat«>n within the geographic map 
structure. Within each Space are or>e or more 
"objects." A user may select and manipulate the 
objects within each Space. In the present embodi- 
ment of Ihe hand-held display device 170 with 

45 touch screen display 37. a user may select objects 
by touching the objects displayed on the touch 
screen display 37. Pointing devk:es other than the 
users Hnger and input devices, other than a touch 
screen, might handle selection and user input in a 

so different manner. 

Spaces 

Each Space in Rgures 2b and 2c has an 
S5 associated background image that contains appro- 
priate imagery for the Space. When a user enters a 
Space, the Space's background image is rendered 
on the display saeen 37. Each background image 
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contains landmarks that help the user identify 
where the user is. For example, the Earth's surlace 
Space 32 of Figure 2b is illustrated as Ihe back- 
ground images in Figure 4 and Figure 5. The 
Earth's surface background image of Figures 4 5 
and 5 resembles a street on Ihe Earth's surface, 
thereby informing a user that the user is at the 
Earth's surface Space of Ihe map struclure. The 
system renders the background image using de- 
saturated or dimmed colors such lhat Ihe back- 10 
ground image can easily be distinguished from 
Selectable objects. The background image for a 
Space may be larger than the display screen 37; 
therefore, the present invention provides a gesture 
for "parking" such that a user may move around is 
within a Space as will be explained later. 

Objects 

Objects are individual graphical icons that are 20 
ren6ere<i within the background images of Spaces. 
Each object is actually an object-oriented program- 
ming type of object containing data methods which 
can be invoked to act on the data. A user can 
select and manipulate any of the objects which 25 
appear within the background image of a Space. 
Each object is brightly colored and outlined in 
black. The bright colors distinguish the objects in a 
Space from the Space s background image that is 
drawn using desaturated colors. When a user 30 
points to an object, the user interface generates a 
"Selectable Object" sound that informs the user 
that the object can be selecjied and manipulated. 
Referring again to Figure 4. in the present exam- 
ple, the houses 47 and 48 are objects that can be 35 
selected arxl manipulated by a user. 

An "anticipation animation" is used in the 
present invention to infonm the user when some 
action is taking place. The anticipation animation is 
a short animation displayed on the display screen 4o 
37 that is generated by the CPU 30. In most cases, 
the anticipation animation consists of enlarging an 
object's icon such that the Icon appears to move 
closer to the user when the object is selected. 

For example, referring to Figure 3. when the 45 
Earth portal 100 is selected, the Earth portal 100 
expands such that it appears to move closer to the 
user. After the Earth portal 100 anticipates, the user 
interfaces move the user to the Earth's surface 
Space 32 that is associated with the Earth portal so 
100 as illustrated in Figure 4. 

When a user selects an object by pointing to 
the object for a predetermined minimum period of 
time, a sound is generated, the object anticipates, 
and the CPU invokes a method within the object 55 
which performs an action associated with the ot>- 
iect. 



There are at least three types of objects that 
exist in the user interface of the present embodi- 
ment: data objects, buttons, and portals. The sim- 
plest object is a data object. Data objects are 
objects associated with a set of data. Buttons are 
objects that provide a means for accomplishing 
functions. Portals provide a method for moving 
between different Spaces. Each type of object will 
be explained individually below. 

Data Objects 

A data object is a simple object used to repre- 
sent some piece of information. For example, a 
data object may represent a color or a television 
channel. The data objects have no methods that 
can be invoked to perform an associated action or 
function when the data object is selected. However, 
a data object can be dropped onto other objects 
that use the infornDation in the data object. For 
example, a television program listing data object 
can be dropped onto a video cassette recorder 
object to program the video cassette recorder to 
record the television program. 

Buttons 

A button object is a special object that has an 
associated action or function. When a user selects 
a tHJtlon. the CPU 30 invokes the action method in 
the button object to cause the execution of the 
action or function associated the button. When the 
button is selected, the CPU also anticipates the 
button and generates a "button action" sound pro- 
viding feedback to the user that the button has 
been selected. Certain buttons provide additional 
visual feedback to inform the user of the results of 
the method invoked by the CPU to perform the 
action associated with the twtton. 

For example, refernng to Figure 8, a television 
power -pn'" button 61 and a television power "off" 
button 62 are displayed. When a user selects the 
television power "on" button 61. the television pow- 
er "on" button 61 anticipates and the CPU 30 
generates the "button action" sound. The CPU 30 
also invokes a method in the txjtton object that 
transmits the necessary information to turn on a 
real television set associated with the television 
control panel illustrated in Figure a 

If the real television set is an "intelligent re- 
mote device", the television set may return a result 
value to the display device infonning the user 
whether or not the televiskm was turned on. The 
user interface may then Bght up the television 
power "on" button 61 and generate a "hum" sound 
to inform the user that the real television was 
actually turried on. 
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Buttons can be copied and moved to different 
Spaces so tfial a user can customize the "world" ol 
tfie user interface. For example, the television pow- 
er "on" 61 and the television power "off 62 but- 
tons could be copied, and the button copies moved s 
from the television control Space to any other 
Space. A user may use the copies of the television 
power "on" 61 and the television power "off" 62 
buttons in the new. Space to control the television. 

70 

Portals 

Portals are objects that provide a simple meth- 
od of moving from the current Space to another 
Space associated with the portal. A user moves is 
from the current Space to another Space asso- 
ciated with a portal by simply Selecting the portal. 
After a user selects a portal, the portal anticipates, 
generates a "movement through portal sound." and 
the CPU invokes a method that moves the user to 20 
the Space associated with the portal. 

For example, referring to the geographic map 
of Figure 2b, to move from the Earth's surface 
Space 32 to the house Space 50. a user simply 
selects the house portal 48. Referring to the Earth's 25 
surface Space 32 as illustrated in Figure 4. a user 
performs the move by selecting the house portal 
48 within the Earth's surface Space 32. After being 
selected, the house portal 48 anticipates, generates 
a movement through portal sound." and then the 30 
user is moved to the house Space 50 as illustrated 
in Figure 6. 

Portals can be duplicated and moved such that 
a user can customize the "world" of the user 
interface. For example, in Figure 7 a television 35 
program schedule portal 71 is Illustrated within the 
living room Space 60. The TV program schedule 
portal 71 is associated with a Space that contains 
TV program schedule information. A user can make 
a copy of the television program schedule porta! 71 40 
and move the copy to another Space where the 
user would like to be able to access the television 
program schedule. The user can then use the copy 
of the television program schedule portal to access 
the same information that could be accessed using 45 
the original television program schedule portal. 
Thus, a user would be able to access the same 
information from different Spaces. 

There are two special portals that are almost 
always available to an operator of the user inter- so 
face: a Wayback portal and an Agent portal. 

The Wayback Portal 

The WaylMCk portal provides a way for a user 55 
to trace back a path that the user has traveled 
atong between different Spaces. The Wayback por- 
tal 45 consists of a small rectangular frame that 



encloses a snapshot of the previous Space along 
the path of Spaces that the user traveled along. 
Since the background image for a Space can be 
larger than what can be seen in a single view, the 
Wayback portal shows a snapshot of the view of 
the previous Space exactly as it appeared when 
the user went through a portal in that previous 
Space. In the present embodiment, the Wayback 
portal is available to the user in every Space in the 
lower right-hand corner of the display screen. If 
desired, the user can move the Wayback portal to 
a new location on the display screen such that the 
Wayback portal will then appear at the new location 
on the screen in every Space. 

To move one step backward along the current 
path, the user Selects the Wayback portal 45. 
When Selected, the CPU 30 anticipates the 
Wayback portal 45. generates a "move through 
Wayback portal" sound, and moves the user to the 
Space pictured within the Wayback portal. In the 
present emt>odiment. the "move through Wayback 
portal" sound is actually the same sound as the 
"move through portal" sound except that it is 
played backwards. By playing the "move through 
portaP sound backward when moving through the 
Wayback portal, the movement backward atong the 
current path is emphasized to the user. 

As users go through portals they leave a trail of 
invisible markers. Each marker represents a visit to 
a Space. The Wayback manages these markers 
and maintains them in Xhe order that the Spaces 
are visited. The markers and the markers' order 
define the path that the user has taken through the 
worb of the user interface. Thus, wfien a user goes 
though the Wayback. the path defined by the mark- 
ers is retraced one marker at a time. Each time the 
user goes through the Wayback to a Space, a 
marker is removed from those currently managed 
by the Wayback. 

For example, referring to geographic map 
structure illustrated in Rgure 2b, if a user starts in 
the outer-space Space 99, moves to the Earth's 
surface Space 32, and then moves to the house 
Space 60; a path will be generated from the outer- 
space Space 99 to the Earth's surface Space 32 to 
the house Space 50. After traversing this path, the 
display screen 37 will display the house Space 50 
as depicted in Figure 6. In the lower right comer of 
the house Space 50 is the Wayback portal 45. As 
illustrated in Figure 6. the Wayback portal 45 dis- 
plays a miniature snapshot of the previous Space 
atong the Space path. In this case, the prevk)us 
Space along the Space path is the Earth's suriace 
Space 32; therefore. Vhe Wayback portal 45 dis- 
plays a miniature snapshot of the Earth's surface 
Space 32 as it appeared when the user went 
through the house portal. H the user then Selects 
the Wayback portal 45. the Wayback portal 45 
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anliclpates. generates the "move through Wayback 
portal" sound, ana then moves the user to the 
Earth's surface Space 32 as depicted In Figure 4. 
At the Earth's surface Space 32 as depicted in 
Figure 4 the Wayback now displays a miniature s 
snapshot of the outer-space Space 99. which was 
Ihe first Space along the path. The Wayback portal 
45 therefore provides a "way back- akmg the path 
of Spaces the user has traveled. 

The present invention implements the to 
Wayback portal 45 as a stack of data structures, 
wherein each data structure contains a pointer and 
a pair of coordinates. The pointer in each data 
structure on the stack points to a Space that the 
user has previously visited. The pair of coordinates ;5 
in each data structure on the stack specifies ex- 
actly where the user was within the Space pointed 
to by the pointer when the user left the Space 
through a portal. Each time the user moves through 
a portal from a particular Space, the user interface 20 
"pushes" a data structure, containing the pointer to 
the Space and the coordinate pair which defines 
exactly where the user was wfthiri the Space when 
he went through the portal, onto the wayback stack. 
The Wayback portal 45 displays a miniature snap- 25 
shot of the Space that is pointed to by the pointer 
in the data structure on the top of the stack. To 
trace back along the path of Spaces, each time the 
user Selects the Wayback portal 45. the user inter- 
face "pops" a data structure off the stack and 30 
moves to the Space indicated by the pointer in the 
data structure. 

The Agent Portaf 

35 

The Agent is an anthropomorphic" character 
that aids the user in navigating around the user 
interface and in dealing with events that occur in 
the "world" created by the user interface. Refening 
to the Earth's surface Space of Rgure 4, the Agent 4o 
41 is illustrated in the lower left corner. The Agent 
41 carries objects around for the user. This carry- 
ing function is represented by a briefcase, which 
the Agent also carries. A user can access objects 
that the Agent is carrying by Selecting the Agent 45 
portal 41 thereby entering an Agent Space asso- 
ciated with the Agent portal 41. When a user Se- 
lects the Agent porta! 41. the Agent anticipates by 
appearing to pull out the briefcase and open the 
briefcase up. The CPU then generates the "move so 
through portal" sound and moves the user to the 
Agent Space. 

To place an object into the Agent's briefcase, a 
user picks up the object and then "Drops" the 
object onto the Agent portal 41. The object 55- 
Dropped onto the Agent portal 41 is then always 
accessible to the user from any Space by entering 
the Agent's Space. The Agent Is discussed more 



fully below in the section entitled "The Agent." 
Object Wheel 

ft is Often necessary that a user make choices 
among an large, arbitrary number of objects. 
Therefore, the user interface must provide both a 
method of displaying a large number of objects 
and a method of alk^wing the user to Select any of 
the displayed objects. 

For example, referring to the television control 
Space 64 illustrated in Figure 8, the user interface 
of the present invention must provide a method of 
displaying an arbitrary number of television chan- 
nels for the user to choose from. In any system, 
screen space is at a premium, so when the user 
must choose among many possible options, the 
optioris must be presented compactly. In the 
present invention, if there are 500 television chan- 
nel objects for a user to select from, presenting 
them all at once wouW not only take up significant 
screen space, but the visual context of the list (i.e., 
why is the user looking at this list in the first place) 
would be lost. Thus, a method for allowing the user 
to browse through the available objects is required. 

The user interlace of the present invention in- 
troduces an "object wheel" to provide a method for 
alkjwing a user to choose from a large group of 
objects. In the object wheel, a few immediately 
accessible objects are displayed in an object wheel 
on the display screen while the remaining objects 
in the group are not rendered on the screen. To 
access the objects not rendered on the screen, the 
user "Spins" the object wheel to move the pre- 
viously inaccessible objects Into view on the dis- 
play screen. 

Referring to Figure 12, a large number of fruit 
objects are available for a user to choose from, yet 
only four fruit objects fit horizontally on the display 
screen 37 at once. To provide the user with access 
to an the fruit objects available, the present inven- 
tion puts the strip of fruit objects onto an "object 
wheel" 38 as displayed in the conceptual diagram 
of Figure 13. A few of the objects on the object 
wt^el 38 are immediately accessible to a user on 
the display screen 37. To access the objects that 
are not displayed on the display screen 37. the 
user rotates the object wheel 38. As the object 
wheel 38 rotates, new objects are brought into view 
and other objects disappear from view as illustrated 
in Figure 14. An operator uses the navigational 
gestures Point. Spin, and Pickup to manipulate the 
object wheel presented on the touch screen display 
37. These three navigational gestures are explained 
in the following section. 

In order to enhance the user's feeGng that a 
real wheel is being spun. ot>|ect wheels obey the 
laws of physics. For example, the object wheels 
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have mass and therefore inertia; object wheels spin 
faster as Ihe user applies more "force • when spin- 
ning the wheel; ot>ject wheels respond to friction 
since a spinning object wheel gradually stops spin- 
ning. 5 

White an object wheel is spinning and the 
objects are moving across the screen, the current 
embocfiment of the user interface generates a "clic- 
king" sound. The "clicking" sound generated by 
the user interface simulates the sound made by io 
detents on a slot machine wheel which forces the 
wheel to stop and be aligned. Like a real wheel, the 
object wheels of the present invention always stop 
aligned, and never stop in fcietween object choices. 
A user can stop a spinning object wheel at any ts 
time by Pointing to a moving object on the object 
wheel while the object wheel is spinning. 

An object wheel can be aligned vertically or 
horizontally on the display screen. When placed 
vertically, the object wheel can t)e spun both up 20 
and down. When placed horizontally, the object 
wheel can be spun both teft and right. 

User Interface Navigational Gestures 

25 

Navigation and object manipulation in the user 
interiace are performed using a screen pointing 
device. In the preferred embodiment, the hand-held 
display device 170 is constructed with a touch 
screen display 37 and the pointing device used is a 30 
user's finger. Alternatively, the user interface may 
be implemented using any other type of screen 
pointir>g device such as a mouse, a trackball, or a 
joystick. This specification, however, will only de- 
scribe the preferred embodiment that uses the 35 
user's finger as a pointing device on a touch 
screen. 

In the preferred embodiment of the present 
invention, the hand-held display device 170 com- 
prises, in part, a touch screen display 37 as illus- 40 
Irated in Figure lb. The touch screen display 37 
senses when a finger has been placed on the 
display screen and identifies where the finger has 
been placed. Users select objects on the touch 
screen and n^anipuiate the objects using a set of 45 
six navigational gestures: Point, Select, Pan, Spin. 
Pickup, and Drop. Each navigational gesture used 
by the user interface is explained individually be- 
low. (Since the navigational gesture names are the 
sanoe as common words, this specification will 50 
capitalize all references to the navigational ges- 
tures.) 

Point 

55 

Point is the most simple gesture in the user 
interface of the present invention. An operator uses 
the Point gesture to pick objects rendered on the 



display screen. For example, if the user interface of 
the present invention requests the user to choose 
one of many objects rendered on Ihe display 
screen, the user Points to an object to choose the 
object. 

A user Points to an object by simply placing a 
finger on the object displayed on the touch screen. 
When a user has successfully Pointed to an object, 
the user interface generates a "Selectable Object- 
sound. The Selectal>le object sound provides feed- 
back to the user indicating that Ihe user has suc- 
cessfully Pointed to the object. 

Select 

The Select gesture is very similar to the Point 
gesture. An operator uses the Select gesture to 
have an object invoke a method to perform an 
function or action associated with the object. For 
example, a Selected porfal invokes a method that 
moves the user to the Space associated with the 
portal and a Selected button invokes a method that 
performs the action or function associated with the 
button. 

To Select an object, a user first Points to an 
object by touching the object with a finger. If the 
user continues to Point to the object for a predeter- 
mined minimum period of time, the object be- 
comes Selected. After a user Selects an object, the 
object "anticipates," generates a sound for user 
feedback, and then a method within the ot^ect is 
invoked. If the object is a button, then the method 
invoked performs a function associated vnth the 
button. H the object is a porfal. then the method 
invoked will move the user to the Space associated 
with that porfal. 

Alternatively, a user can Select an object on 
the display screen by quickly Pointing to the object 
twice in succession. When a user Selects an object 
by quickly Pointing to the object twice, the object 
does not anticipate. Instead, the user interface only 
generates the appropriate feedback sound and then 
performs the action associated with the object. This 
provides the user a method of Selecting an object 
quickly. 

Pan 

To move around within a Space a user uses 
the Pan navigational gesture. A user Pans though a 
Space by placing a finger on the t>ackground im- 
age of a Space displayed on the touch cfisplay 
screen 37 and slicfing the finger tn the desired 
direction of movement. The user interface of the 
present invention responds by generating a "Pan 
Active" sound and moving background image of 
the Space and all the objects wihin the Space in 
the opposite direction of the finger's movement. By 
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moving the Space in the opposite direction, the 
user interface simulates movement of the users' 
point of view in the other direction. 

Figures 4 and 5 illustrate the use of the Pan 
gesture. If a user places his finger on the back- s 
ground of Figure 4 and then slides the finger on 
the touch screen to the right, the view wilt cor- 
respondingly pan to the right by moving the back- 
grourid image of the Space and all the objects 
within the Space to the left. Figure 5 illusUates the w 
result of the Pan right operation where house 48 
has moved from the right side of the display 
screen over to the left side of the display screen. 

The background image for each Space in the 
present lnventk>n can be arbitrarily large. If the 75 
background image of a Space has finite size, the 
user interface of the present invention provides 
feedback that the furthest extent of the background 
image has been reached. When a user Pans in a 
direction all the way to the end of a background 20 
image, the Pan gesture wll not move the user any 
further in that direction. To inform the user of this 
fact, the user interface generates an "end of ex- 
tent" sound instead of the "Pan Active" sound 
when a user attempts to Pan past the end of a 25 
background image. The "end of extent" sound 
sounds like a "thud" made when something bumps 
into a wall. The "end of extent" sound thereby 
informs the user that the user has hit the end of the 
background image and can go no further. 30 

Spin 

An operator uses the Spin gesture to browse 
through several objects placed on an object wheel. 35 
When a user Spins an object wheel, the visible 
objects on the object wheel move across the dis- 
play screen and disappear off one end of the 
screen and previously invisible objects move onto 
the display screen at the other end, 40 

A user Spins an object wheel by placing a 
finger on the object wheel and then nrK)ving the 
finger across the display screen in a direction 
parallel to the ot>ject wheePs movement. Figure 15 
illustrates the Spiri gesture. In Figure 15, a user 45 
first places a finger on the object wheel (as shown 
by the asterisk 96). The user then slides the finger 
across the object wheel (as shown by the arrow 
97). This finger gesture causes the object wheel to 
Spin in the direction of the finger's movement. The so 
faster the user slides the finger across the object 
wheel on the display, the faster the object wheel 
spins. Once the object wheel is spinning, the ob- 
jects on the object wheel appear at one end of an 
object wheel, move across the display screen, and 55 
then disappear off the other end of the object 
wheel similar to figures wlu'ch appear on a stot 
machine wheel. 



Pickup 

The Pfckup gesture is used to pickup an object 
located within a Space or on an object wheel such 
that a user can carry the object around the "world" 
defined by the user interface of the present Inven- 
tion.. In the present embodiment, different methods 
are used to Pickup an object from a Space or an 
object wheel. Each method will be explained in- 
dividually below. However, it will be appredaled 
that the same gesture couW be used to pkkup 
objects in both situations. 

The gesture used to Pickup an object from an 
object wheel is illustrated in Figure 16a. Figure 
16a illustrates an object wheel 91 with a number of 
fruit objects to choose from. To Pickup one of the 
fruit objects disposed on the object wheel 91. a 
user -first places a finger on the object (as des- 
ignated by the asterisk 92) and moves the finger in 
a direction perpendicular to the motion of the ob- 
ject wheel. The movement of the finger in a direc- 
tion perpendicular to the motion of the object wheel 
is illustrated by arrow 93. The perpendicular finger 
motion "tears" the object off the object wheel such 
that the user can move the object. When the object 
comes loose from the object wheel, the user inter- 
face generates a "tearing" sound that informs the 
user that the object has been removed from the 
object wheel. When an ot>iect has been picked up. 
the object casts a shadow onto the background 
image of the Space and moves the object ak>ng 
with the user's finger. 

The gesture used to Pickup an object from a 
Space is illustrated in Figure 16b, Figure 16b 
illustrates a Space with an object in the center. To 
Pickup the object, a user circles the object with the 
finger as illustrated by arrow 203 and then touches 
the object as illustrated by asterisk 201. When the 
object comes loose from the Space, the user inter- 
face generates the tearing sound that informs the 
user that the object has been picked up. After an 
object has been picked up from a Space by a user, 
a ghost image of the object remains at the object's 
original location. The gfiost image of the object is 
drawn in desaturated cotors indicating to the user 
that the ghost object cannot t>e Selected. 

After a user has picked up an object, eitfier 
from an object wheel or a Space, the user interface 
generates a "carrying object" sound that informs 
the user that an object is being carried. The user 
can move the carried ol^ct within a Space, from 
Space to Space, or Drop the carried object onto 
arK3ther object. 

Drop 

The final gesture used to interact with the user 
interface of the present invention is Drop. After a 
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user has used the Pickup gesture to pick up an 
object from a Space or from an object wheel, the 
user must continue to keep the finger pressed 
against the screen to carry the object. The user 
moves the carried object by sliding the finger s 
across the touch screen display system. After mov- 
ing the carried object to a desired location, the 
user then uses the Drop gesture to place the object 
at the desired k)cation. To drop the object the user 
lifts the finger off the touch screen display. When lo 
the user Drops an object, the object falls into the 
background image of a Space as if pulled by 
gravity, and the object no longer casts a shadow. 

A user of the present invention uses the Drop 
gesture to perform a number of different functions. ;5 
The Drop gesture can be used to copy an object 
by picking up the object and Dropping the object 
into a portal such that a copy of the object is 
created in the Space associated with the portal. 
The Drop gesture can l>e used to move an object 20 
to a new location by picking up the object and 
Dropping the object at the desired tocation. The 
Drop gesture can also be used to provide a data 
object containing information lo another object 
which uses the information to perform a function. 25 
Each different use of the Drop gesture will be 
explained below. 

To make a copy of an object using the Drop 
gesture, a user must first Pickup the object and 
then Drop the object onto a portal. An object that 30 
has been picked up is carried over the portal by 
sliding the finger across the screen. Once the car- 
ried object is over the portal, tf»e portal becomes 
highlighted such that it provides feedback to the 
user signaling that the carried object is over the 35 
portal. In the present emixidiment. the portal emits 
a green glow. To Drop the carried object into the 
portal, the user lifts the finger away from the touch 
screen display. The object is then displayed such 
that it passes through the portal with an appropriate 40 
animatk)n. During the animation, the CPU 30 gerh 
erates a "sucked through portal" sound effect. 
When the object has completely disappeared 
through the portal, the ghost image of the object 
drawn in desaturated colors in the present Space 4S 
changes back into a brightly colored object. Tfiis 
provides visual feedback to the user that the origi- 
nal object remains in the present Space even 
though a copy of the object has t>een sent through 
the portal to another Space. 50 

Users can move objects either within a single 
Space or frpm one Space to another. To move an 
object within a single Space a user first uses the 
Pickup gesture to pick up an object, moves the 
object to the desired new location, and then Drops ss 
the object using the Drop gesture. After the object 
rs Dropped, the ghost image of the object merges 
with the Dropped object at the new location to form 



a single brightly colored image of the object at the 
new location. This informs the user that the object 
has been moved to a new location and that a copy 
has rK)t been made. 

Moving an object to aru>ther Space is accom- 
pfished by carrying the object through a portal to 
arvjther Space. To move an object lo another 
Space, a user most first Pickup the object and then 
carry the object over a portal. Once the carried 
object is over the portal, the portal becomes high- 
lighted such that it provides feedback to the user 
signaling that the carried object is over the portal, 
and the portal emits a green gtow. To move the 
carried object through the portal, the user holds the 
object over the portal for a predetermined mini- 
mum amount of time. The portal eventually antici- 
pates moving the user and the carried object to the 
Space associated with the portal. The ghost image 
of the carried object drawn in desaturated cotors is 
also moved atong. with the user, and the carried 
object to the destination Space. When the user 
Drops the carried object by Bfting his finger off of 
the touch screen display system, the carried object 
and the ghost image merge into a single brightly 
colored image of the object. This provides feed- 
back to the user that the object has been moved to 
the Space and has not l)een copied. 

The Drop gesture can also t>e used to provide 
information to an object. A user provides informa- 
tion to an object by Dropping an object containing 
data (i.e. a data object) onto the object which will 
use that data. 

To provide rnformalion to an object, a user 
must first Pickup a data object that contains the 
necessary information. The user then carries the 
data object over the object which can utilize the 
information. When the carried data object is over 
the object that can utilize the information, the ob- 
ject that can utilize the information l:>ecomes high- 
lighted if the carried data object is of the correct 
class. By becoming highlighted, the object that can 
utilize the information provides feedback to the 
user that the carried data object is of the correct 
class. To Drop the data object into the object that 
can utilize the information, the user lifts his finger 
used to carry the data object away from the touch 
screen. The data object is then passed through and 
into tfie portal with appropriate animation and 
sound effects. When the data object has com- 
pletely disappeared into the receiving object which 
can use the information, the receiving object then 
performs the appropriate functk>n on the data pro- 
vided. 

For example, the Drop gesture can be used to 
provide television program information to the video 
cassette recorder object which is associated with a 
physfcal video cassette recorder, such that the 
video cassette recorder programs itself to record 
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the television program. To program the video cas- 
sette recorder, a user first uses the Pickup gesture 
to pickup a data object containing television pro- 
gram information. The user then carries a television 
program data object over the video cassette re- 
corder object. The video cassette recorder object 
then modifies its appearance to indicate it is a 
"program video cassette recorder* mode. This in- 
forms the user that the video cassette recorder 
"understands" the information that is over it (i.e., 
the video cassette recorder will be programmed to 
record the television program if the user Drops the 
television program data object onto the video cas- 
sette recorder). 

When the user Drops the television program 
object into irv^; video cassette recorder, the video 
cassette recorder object recognizes that the televi- 
sion program data object contains information it 
can use. The video cassette recorder then extracts 
the time and channel information from the televi- 
sion program data object such that the video cas- 
sette recorder object programs itseH to record the 
television program. 

A Tour of the Present Embodiment in Opera- 
tion 

To best convey the look and feel of the user 
interface of the present invention, a "tour" through 
a sample environment is provided with reference to 
Figures 3-10. The sample environment explored 
in the tour is based upon . the geographic map 
structure of Spaces diagrarnmed in Figures 2b 
and 2c. The tour through the sample environment 
descrit>ed herein is not intended to limit the scope 
of the present invention, but instead to provide an 
example of one possible emtxKJiment of the 
present invention. Due to the limitations of a written 
Specification in descriljing the present invention's 
animated user interface, the reader is encouraged 
to view a video tape entitled "Green Project User 
Interface,- submitted by the applicant concurrently 
with the fi0ng of the application on which this 
patent is based. 

Referring to Figure 3, upon first entering the 
user interface of the present invention, the user 
interface irvtially positions the user in an outer- 
space Space 99. This Space is Illustrated in the 
geographic map structure of Spaces diagrammed 
in Figure 2b. Although, the user is initially placed 
in the outer-space Space 99, the user interface will 
later start the user at the last Space the user was 
in when the user quit using the handrheld display 
device. Referring again to Rgure 3. at the outer- 
space Space 99, the user interlace renders a 
brightly colored Earth portal 100 against a dark 
outer-space background image on the display 
screen. 



Soon after entering the user interface, an out- 
line of a hand 31 appears on' the display screen to 
leach the user how to navigate around the user 
interface. The hand 31 that appears on the display 
5 screen is the hand of the Agent. The Agent's hand 
31 extends a Hnger to touch the screen and then 
slides the finger across the screen causing the 
Space and objects on the screen display to Pan 
and the user interface to emit the "Panning active** 
10 sound. Referring to Rgure 3. the outline hand 31 
of the Agent is shown moving across the screen. 
By touching the display screen and moving a fin- 
ger across the display screen, the Agent's hand 31 
demonstrates to a new user how to use the Pan 
IS gesture to move around the view of a Space. 

After demonstrating how to use the Pan ges- 
ture to nwve around in a Space, the hand 31 of the 
Agent next demonstrates how to use the Point and 
Select gestures. The Agent demonstrates how to 

20 use the Point gesture to Point to an object by 
placing a finger on the Earth portal 100. After the 
Agent Points to the Earth portal 100, the user 
interface emits the "Selectable Object" sound to 
provide feedback which informs the user that the 

25 Earth portal 100 is a Selectable Object, The Agent 
then demonstrates how to use the Select gesture 
to Select an object by continuing to Point to the 
Earth portal 100. After the Agent's hand 31 Points 
to the Earth portal 100 for the predetermined mini- 

30 mum amount of time, the Earth portal 100 antici- 
pates and the CPU generates the "move through 
portal" sound. The Earth portal 100 anticipates by 
expanding rapidly such that the Earth portal 100 
appears to move closer to the user. The screen 

35 then fades and a new background image appears 
displaying the Earth's surface Space 32 associated 
with the Earth portal 100. as illustrated in Figure 4. 

At the Earth's surface Space 32. the Agent 
portal 41 appears to "jump" onto the touch screen 

40 display 37 and moves to the tower left comer of 
the display screen. The Agent 41 will always be 
accessible to the user because the agent appears 
in every Space, except the Agent's Space. H de- 
sired, the user can move the Agent from the lower 

45 left corner to a different kx:aiion on the display 
screen. This movement of tf>e Agent will change 
the Agent's location In every Space. 

Referring to Figure 4. in the lower right corner 
of the display is the Wayback portal 45. The 

50 Wayback portal 45 displays a framed snapshot of 
the outer-space Space 99 from Figure 3 which the 
user just came from. If the user Selects the 
Wayback portal 45. the user interface of the 
present invention moves the user back atong the 

55 traveled path to the Space illustrated within the 
Wayback porta! 45. For example, if the user de- 
sires to return to the outer-space Space 99 dis- 
played in the Wayback portal 45, the user simply 

15 
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Selects the Wayback portal 45. 

The Earth*s surface Space 32 depicted in Fig- 
ure 4 is comprised of a number of houses arxJ 
community buildings such as stores, banks, 
schools, etc. Each building in the Earth's surface 
Space 32 represents a portal that can be Selected 
by the user to move the user to a Space asso- 
ciated with the portal. For example, in the Earth's 
surface Space depicted In Figure 4. the first house 
47 and the second house 48 are portals which can 
be Selected by the user. If the user Selects either 
the first house 47 or the second house 48, the user 
will t>e moved to an associated house Space. 

While within the Earth's surface Space de- 
picted in Rgure 4. the user can move around in 
the Space by using the Pan gesture on the touch 
screen as demonstrated by the Agent's hand 31 in 
Figure 3, To Pan. the user simply places a finger 
on the screen and moves the finger in the direction 
the user wishes to move. The user interface re* 
spends to the user's finger motion by generating 
the "Panning Active" sound and moving the fc>ack- 
ground image and the objects on the screen in the 
opposite direction of \he finger's movement, there- 
by bringing other objects into the view. 

For example, if a user viewing the Earth's 
surface Space 32 as depicted in Figure 4 wishes 
to move to the right, the user simply places a 
finger onto a background area of the Earth's sur- 
face Space 32 and slides the finger to the right. 
The user interface reciprocally responds to the 
user's rmger motion by moving the Space's back- 
ground image and the objects iri the Space to ttie 
left. Figure 5 depicts the view ot the Earth's sisr- 
face Space 32 after the user has placed a finger 
onto a background area of Figure 4 and slid the 
finger to the right. A first house portal 47 displayed 
in Figure 4 has moved out of the view and is not 
visible in Figure 5. A second house portal 48 
displayed on the right side of the view in Figure 4 
has moved to the left side of the view in Figure 5 
and now is only partially visible. A l>ank portal 49 
that was not visible in Figure 4 has moved into 
view from the right as illustrated in Figure 5. The 
Agent 41 and Wayback portal 45 remain in the 
view In the same lower left and fcDwer right posi- 
tions respectively. 

When the user Selects any portal from the 
Earth's surface Space 32 as depicted in Figures 4 
and 5. the Selected portal anticipates to provide 
feedback to the user and the user intertace moves 
the user to a Space associated, .with the Selected 
portal. For example, when a user at the Earth's 
surface Space 32 depicted in Figure 4 Selects the 
first house portal 47, the house portal 47 antici- 
pates and then the user intertace moves the user 
to a house Space associated with the house portal 
47. Figure 6 illustrates a house Space 50 asso- 



ciated with the house portal 47 of Figure 4. 

The house Space 50 displayed In Figure 6 
consists of a house with the outer walls removed 
such that the individual rooms inlside Ihe house are 
5 visible. Ail of the rooms inside the iKHise, such as 
the living room 54. bedroom 53. bathroom 55. attic 
52. and kitchen 56 depicted in Figure 6. are por- 
tals that can be Selected by the user. As illustrated 
in Rgure 6, the Wayback portal 45 contains a 

10 miniature snapshot of Ihe Earth's surface Space 
32, as it appeared when the user left the previous 
Space. Thus the Wayback portal 45 depicted in 
Figure 6 not only contains a miniature view of the 
Earth's surtace Space, but a view of that portion of 

fs the Earth's surtace Space 32 that was in view when 
the user Selected the house portal 47 of Rgure 4. 
To move back up to the Earth's surface Space of 
Figure 4. the user Selects the Wayback portal 45 
that contains a snapshot image of the Earth's sur- 

20 face Space. 

Referring to Figure 6, if the user Selects the 
living room portal 54, the living room portal 54 
anticipates, the "move through portal" sound is 
gerierated, and the user intertace moves the user 

25 to a living room Space associated with the living 
room portal 54. Referring to Figure 7, an embodi- 
ment of a living room Space 60 is depicted. Within 
the living room Space 60 of Figure 7 is a television 
73, a video cassette recorder 75, and a television 

30 program schedule book 71. The television 73. vid- 
eo cassette recorder 75. and television program 
schedule book 71 are each portals that can be 
Selected by the user. The television 73 and the 
video cassette recorder 75 portals are associated 

35 with a real television and video cassette recorder 
respectively. 

When a user Selects the television portal 73 
displayed within the living room Space 60 of Fig- 
ure 7. the television portal 73 anticipates and the 

40 user interface displays a graptiical user interface 
associated with the real television. Figure 8 depicts 
the' television control Space 64 which comprises 
the television's graphical user interface in the 
present embodiment. The objects available in the 

45 television Space 64 include a green "on" button 
61. a red "off" button 62, and several television 
channel Ixittons 66 - 69. The television channel 
buttons 66 - 69 are on a horizontally aGgned object 
wheel 63 such that a user can Spin the object 

50 wheel 63 left or right in order to reveal additional 
television channel buttons that are not currently 
displayed on the display screen. 

The on button 61. off button 62, and television 
channel button 66 - 69 objects available in the 

55 television Space 64 are not portals that give access 
to other Spaces in the Space map of Rgure lb. 
Instead, the on button 61. off button 62. artd televi- 
sion channel buttons 66 - 69 are buttons that are 
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associated with functions. For example, the green 
"on" button 61 and red "oft" button 62 are asso- 
ciated with the functions "turn the television on* 
and "turn the television off respectively. When a 
user Selects either the on or off button, the button s 
anticipates to provide feedback to the user that the 
button was Selected. The hand-held display device 
then transmits ttie appropriate commands that turn 
the television on or off. Similarly, if the user Selects 
any of the television channel buttons 66 • 69. the w 
television channel button anticipates and the hand- 
held display device transmits the appropriate com- 
nriands to the real television thereby tuning the 
television to the Selected channel. The process of 
transmitting commands to the physical television is 
set is explained in detail in the following section. 

Referring back to Figure 7. if a user Selects 
the television program schedule book portal 71. the 
television program schedule book portal 71 antici- 
pates, and the user interface moves the user to the 20 
television program schedule Space. Figure 9 de- 
picts the television program schedule book Space 
of the present embodiment. In the television pro- 
gram schedule tjook Space of Rgure 9, the user is 
presented with a two dimensional array of televi- ss 
sion program listirigs. The array organizes the tele- 
vision program listings vertically by television chan- 
nel and horizontally by lime. Each television pro- 
gram listing is a portal that can be Selected by the 
user. Each television program listing can also be 30 
carried by the user and Dropped onto the video 
cassette recorder portal such that the hand-tield 
display device programs ibp video cassette re- 
corder to record the television program. 

The television program schedule book Space 35 
of Rgure 9 is actually organized as a pair of 
connected vertically oriented object wheels. Refer- 
ring to Figure 11, a conceptual diagram of the 
television program schedule book Space is illus- 
trated. The television channels of Rgure 9 are 40 
placed on a first object wheel 72 as illustrated in 
Figure 11. The television program fistirtgs are 
placed a second object wheel 85. The two object 
wheels 72 and 85 are connected such that if the 
user Spins either object wheel, the other object 45 
wheel will spin at the same rate such that the 
television channels and television program listings 
are always aligned. 

Each television program listing on the televi- 
sion program schedule object wheel 85 is a portal. so 
When a user Selects one of the television program 
listing portals, the television program listing portal 
anticipates and the user interface moves the user 
to a television program informatk)n Space. Figure 
10 iBuslrates a typical television program infornna- ss 
tion Space. The television program information 
Space includes a ckxk 81. a date 82 and television 
channel 83 whk:h inform the user when the televi- 



sion program is on and what channel the television 
program is oh. The television program information 
Space also irKludes a description of the television 
program as illustrated in Rgure 10. 

The Agent 

The Agent is an anthropomorphic character 
designed to help a user of the present invention in 
a number of ways. The Agent assists users by 
providing help, providing state information, alerting 
the user to certain events, and acting as a reposi- 
tory. Each fufKrtion performed by the Agent will be 
explained indivkfually. 

Help 

As previously described in the tour of the 
present embodiment, with reference to Figure 3. 
when a user first enters the user interface of the 
present invention, the Agent demonstrates how to 
use the Point gesture and the Select gesture. Fur- 
thermore, the Agent demonstrates how to use the 
Pan gesture to move around within a Space. The 
Agent can also demonstrate how to use other fea- 
tures of the user interface as necessary. 

To further aid users, the Agent carries a help 
manual in its briefcase that is available to the user 
at any time. To access the help manual, the user 
simply Selects the Agent portal. The Agent antici- 
pates by providing an animation of the Agent open- 
ing up its briefcase. The user then Selects the help 
manual from within the Agent's briefcase. 

State and Mode Information 

The Agent is also used to express the current 
state or mode of the user interface. To express the 
current state or mode the Agent changes its ctoth- 
ing, changes its expression, or changes the props 
it is carrying. 

For example, if the user interface of the 
present invention is placed into a diagnostic mode, 
the clothing that the Agent is wearing is changed to 
convey this fact. Referring to Figure 17a. the 
Agent is shown in a doctor's uniform to symtwiize 
that the user interface is in a diagnostic mode. 

Figure 17b provides another example of how 
the Agent's appearance can convey state informa- 
tion. When the user is in a writing state, the Agent 
is given a pencil to symbolize that the user inter- 
face is in a writing state. 

Alerts 

While a user is using the hand-held display 
device of the present invention, certain events may 
occur that require the user to be informed imme- 
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diately. Such events are usually referred to as 
alerts. For example, if the hand-held display device 
is coupled to an electronic messaging service, the 
hand-held display device should alert the user 
when a message has been received that is ad- 5 
dressed to the user. 

In the user interface of the present invention, 
the Agent is used to obtain the user's attention in 
order to inform the user about any alerts. To get 
the user's attention the Agent waves its hand. Addi- to 
tlonalty. if the alert is very important, the Agent 
may wave both arms or produce sounds to further 
attract the user's attention. 

For example, when an event occurs that re- 
quires the user's attention, the Agent t>egins wav- is 
ing as illustrated in Figure 17c. To explain what 
caused the alert, the Agent's appearance may be 
further altered as demonstrated in the previous 
section in order to convey information about the 
specific alert. For example, when an urgent mes- 20 
sage arrives, the Agent can wave with one hand 
\Yhile holding a telegram in the other hand. 

Since alerts are generally n>ore important than 
state or mode information, alerts will take prece- 
dence over state or mode information. Therefore, if 25 
the image of the Agent Is currently being used to 
convey state or mode information when an impor- 
tant event occurs, the image of the Agent will 
change to get the user's attention and inform the 
user about the alert. For example, if the Agent is 30 
holding a pencil while in writing mode, as illustrated 
in Figure 17b, when an urgent message arrives, 
the Agent will immediately change to the waving 
animation as illustrated in Figure 17c to inform the 
user about the message. 35 

Remote Device Control 

The hand-held display device of the present 
invention is designed to control remote devices. 40 
Accordingly, the user interface defined in the pre- 
vious section for the hand-held display device is 
designed to be an ideal graphical'user interface for 
controlling remote devices. For example, to control 
a device in the user's living room, a user first 45 
moves to the living room Space 60 of the user 
interface as illustrated in Figure 7. The user then 
Selects an ob/ect corresponding to a device that 
the user wishes to control. For example, if the user 
desires to control a television, the user Selects the so 
television portal 73 associated with a real televi- 
sion. When the user Selects the television portal 
73. the television portal 73 anticipates and then 
displays a graphical user interface for controlling 
the physical television in the user's living room. 55 
Rgure 8 illustrates one possible embodiment of a 
set of television controls for controlling a television. 
The user then controls the television using the set 



of television controls displayed on. the hand-held 
display device. 

In order for a remote device to be controlled by 
the hand-held display device, the hand-held display 
device must have certain information about the 
remote device. Specifically, the hand-held display 
device must be provided with an object-oriented 
programming type of object that is associated with 
the remote device. The obiect comprises a plurality 
of methods including a method for displaying the 
object on the display screen of the hand-held dis- 
play device and a method for displaying a graphi- 
cal user interface for the remote device. 

To simplify the operation of the present inven- 
tion. each compatible remote device automaticaliy 
transmits the required object-oriented programming 
type of object to the hand-held display device. 
When a remote device transmits the object to the 
hand-held display device, the hand-held display 
device invokes the method for displaying the object 
on the display screen. The user can then move the 
object to an appropriate Space. If the user Selects 
the object, the Selected object anticipates and then 
the method for displaying the graphical user inter- 
face is invoked. The method displays the remote 
device's graphical user interface on the hand-held 
display device's touch screen display. The user 
can then control the remote device by interacting 
with the remote device's graphical user interface 
displayed on the touch screen display. 

The hand-held display device of the present 
invention automatically obtains a graphical user in- 
terface from two types off compatible remote de- 
vices. The first type of compatible renrurte device is 
the intelfigent remote device with a dedicated com- 
puter control system. The second type of compati- 
ble remote device is a simple remote device that 
can only respond to simple coded commands. 

The harKJ-heW display device interacts differ- 
ently with the two types of compatible remote 
devices. The hand-held display device commu- 
nicates with intelligent remote devices using peer- 
to-peer communication with remote method invoca- 
twn for remote device control. The hand-tiekf dis- 
play device communicates with simpte remote de- 
vices using only simple control codes. However, 
from the user's perspective the differences be- 
tween the intelligent remote devices and the simple 
remote devices cannot be seen, and the user's 
interaction with the graphical user interface is iden- 
tical for both types of remote devices. The inn- 
piementation detail of whether a user's actions 
cause a remote method invocation, or transmission 
of control codes through some communication 
mechai^m is hidden from the user. 

This section wll explain how the two types of 
compatibie remote devices transmit information to 
the hand-held display device. Furthermore, a third 
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type of remote device will explained. The third type 
of device consists of any electrical device that can 
be controlled by coded Infrared signals. 

Within this section, reference will be made lo 
object-oriented programming type objects. Each s 
object-oriented programming type of object con- 
tains data and methods that can be invoked to 
perform functions. To simplify the terminology 
used, the term "program object" will be used when 
referring to an object-oriented programming type of lo 
object, to eliminate confusion, the objects dis- 
played on ihe touch screen display in the user 
interface disclosed in the previous user interface 
will be referred to as "disptay objects" in this 
section. ,^ 

Peer to Peer Communication With Remote 
Method Invocation 

In the present invention, a remote device is 20 
defined as an "intelligent remote device' rt that 
remote device contains a microprocessor, a com- 
munication system, and a sufficient amount of 
memory such that it can invoke methods in a 
program object. Figure 1c illustrates, in block dia- 25 
gram form, the components in an intelligent remote 
device. The hand-held display device of the 
present invention communicates with intelligent re- 
mote devices in a peer-to-peer fashion. 

The intelligent remote device has an asso- 30 
cialed graphical user interface comprising touch 
screen control elements such as dials, switches, 
sliders, etc. The intelligent re^note device's graphi- 
cal user interface is defined wiOiin a user interface 
program object 303 stored in the intelligent remote 35 
device's memory 404. When an intelligent remote 
devrce is first turned on, the Intelligent remote 
devfce automatically transmits its user interface 
program object 303 to the hand-held display de- 
vice. The hand-held display device can then dis- 4o 
play the user interface for the inteirigent remote 
device as defined by the user interface program 
objecl. 

When a user interacts with the intelligent re- 
mote device's graphical user interface as displayed 45 
on the hand-held display device, the hand-held 
display device invokes methods in a device driver 
program object 301 within the intelligent remote to 
control the remote device. Thus the software that 
actually "controls" an intelBgent remote device is so 
kxated within the intelligent remote device itself. 

The procedure of transmitting the user inter- 
face object 303 from an intelligent remote device to 
the hand-held display device will be disclosed with 
reference to Figures 18a - 18d. 19a - 19d and the 55 
ftowchart in Figure 20. Figures 18a - 18d illus- 
trate, on a high level, how the intelligent remote 
device 150 and the hand-held display device 170 



of the present invention communicate with each 
other. Figures 19a - 19d illustrate, in conceptual 
form, how the user interface program object 303 in 
Ihe intelligent remote device 150 is transmitted to 
the hand-held display device 170. 

Referring to step 101 of Rgure 20. when a 
new intelQgent renwte device 150, such as the 
intelligent thermostat control device in Rgure 18a. 
is first turned on, the new intelligent remote device 
150 broadcasts across the communications net- 
work 160 that it has a user interface program 
object to export. As set forth in the hardware sec- 
tion, the communication on the communicaUons 
nehwork 160 can take place using radio waves, low 
power, cellular, infrared signals, or electrical signals 
on existing power lines in the case of an Echelon® 
base LON® system. In step 102. a hand-held dis- 
play device 170 receives the interface export mes- 
sage. The hand-heW cfisplay device 170 signals 
back across the communications network 160 to 
the new intelligent remote device 150 requesting 
the new intelligent device 150 to begin Uansmitting 
the user interface object 303. 

The Intelligent remote device 150 then trans- 
mits the user interface objecl 303 lo the display 
device 170 as specified in step 103 of Figure 20 
and illustrated in Figure 18a. Referring to Figure 
19a. the intelligent remote device 150 also trans- 
mits a pointer (as indicated by dotted lines) to a 
device driver program object 301 within the intel- 
ligent remote device 150. The user interface pro- 
gram object 303 defines how the user interface for 
the intelligent remote device 150 shoukJ appear on 
a display screen. The user interface program ob- 
ject 303 invokes methods in a device driver pro- 
gram object 301 to handle user interface inter- 
actions. The device driver program object 301 con- 
tains methods that control the actual intelligent 
remote device 150. Referring to Figure 19b. the 
display device 170 now has a copy of the intel- 
ligent device's user interface program object 313 
and a device driver program object 311 pointer. 

After receiving the user interface program ob- 
ject 313. the hand-held display device 170 invokes 
a method in the user interface program object 313 
that renders a display object in ihe current Space 
as stated in step 104. The new intelGgent devkre 
150 also registers with a remote name senrer con- 
nected lo the communk:ations network 160. 

When a user Selects the display object asso- 
ciated with the new intelKgent remote device 150, 
the hand-heW display device 170 invokes a method 
in the user interface program object 313 as stated 
in step 105 (the object with an invoked method is 
shown in bold). The method invoked in the user 
interface program object 313 displays the remote 
inteingent device's graphical user interface on the 
display screen 171 of the display device 170 as 
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illustrated in Figure 18b. A user can interact with 
the remote intelligent device's graphical user inter- 
face displayed on the display device 170 as stated 
in step 106. 

The user controls the intelligent remote device 
150 by interacting with the graphical user interface 
displayed on the hand-held display device 170. To 
resolve the user's interactions, the user interface 
program object 313 requires control methods in the 
intelligent remote device's device driver program 
object 301. To access the control methods in the 
intelligent remote device's device driver program 
object 301, the hand-held display device 170 uses 
the device driver program object pointer 311. The 
device driver program object pointer 311 invokes 
the methods in the real device driver program 
object 301. The intelligent remote device's device 
driver program ot:>ject 301 is invoked by sending a 
message over the communication network 160 as 
illustrated in Figures 18c and 19c (again, the 
object with an invoked method is shown in lx)ld). 
The methods invoked in the intelligent remote de- 
vice's device driver program object 301 control the 
intelligent device 150 as stated in step 108. 

Finally . in step 109. any return values or feed- 
back generated by the device driver program oiy 
ject 301 in intelligent remote device 150 are sent 
back to the hand-held display device 170 over the 
communications network 160 as illustrated in Fig- 
ures 18d and 19d. The user interface program 
object 313 changes the graphical user interface on 
the display screen 171 in response to the return 
values or feedback received from device driver 
program object 301 in the intelligent remote device 
150. 

A robust communk:ation protocol across the 
communications network 160 is required to com- 
municate remote method invocations between the 
harKJ-held display device 170 and the intelligent 
remote device 150. Both the hand-held display 
device 170 requestirtg the method invocation and 
the intelligent remote device 150 that executes the 
method must t>e able to handle a txoken commu- 
nication path. Therefore, both the hand-held display 
device 170 and the intelfigent remote device 150 
have a decficated remote procedure call (RPC) 
manager that handles the renrwte method invoca- 
tion. Referring to Figures 19a - 19d. the renrwjte 
procedure call manager 309 transmits the remote 
method invocation request for the hand-hekj dis- 
play device 170. On the other side, a server remote 
procedure call manager 307 processes the re- 
ceived remote method invocation in an intelligent 
remote device 150. 

Figures 21a and 21b illustrate state diagrams 
used to control the remote procedure call (RPC) 
managers that handle the remote method invoca- 
tion on both the hand-held display device 170 side 



and the remote device side. Taken together, state 
diagrams in Figures 21a and 21b disclose a hand- 
shake used between the hand-heki display device 
170 and an intelligent remote device 150 to ensure 
5 reliable communications. 

Figure 21a illustrates a state machine for the 
remote procedure call manager 309 in the hand- 
held display device 170 that handles the display 
device's skie of a remote method invocation. In the 

10 present invention, the user interlace program object 
313 in the hand-held display device 170 requests 
remote method invocations to the device driver 
program objects 301 located in remote intelligent 
devices. Referring to Figure 21a. when a user 

75 interface object requests a remote method invoca- 
tion, the hand-heM display device's RPC manager 
enters the "ready to transmit" state 403. The dis- 
play device's RPC manager then sets an error 
countdown timer and transmits a packet containing 

20 a message with a metfiod selector and associated 
data. After transmitting the packet, the display de- 
vice's RPC manager enters a "waiting for repfy" 
state 405. If the proper reply message is sent t>ack, 
the method invocation is complete and the reply 

25 message is analyzed at state 407. 

Remote method invocations do m>t always 
complete so easily. If the device driver program 
object 301 in the intelligent remote device 150 
needs more time to complete the method, it will 

30 send back a message requesting more time. The 
display device's RPC manager responds by mov- 
ing to state 409 where it increases the timeout 
value. If the display device's RPC manager does 
not receive, a reply by the time the enor countdown 

55 timer expires, the display device's RPC manager 
moves to an error recovery state 411. If the error 
countdown timer expires less than a predetermined 
numl)er of limes, the display device's RPC man- 
ager returns to the "ready to transmit" state 403. ff 

40 the error countdown timer reachies a predetermined 
limit, the display device's RPC manager alsandons 
the remote method invocation and enters a failed 
state 413. The user is then usuaRy informed of the 
communications problem. 
- 45 Figure 21b illustrates a state machine for a 
remote procedure call manager 307 in an intelfigent 
remote device that handles the remote device's 
side of a remote method invocation. The remote 
device's RPC manager 307 starts in a "waiting to 

50 receive" state 451. When a packet is received, the 
remote device's RPC mar\ager analyzes the packet 
at state 453. If the received packet contains a new 
message requesting a method invocation, the re- 
mote device's RPC manager gives the packet to an 

55 object that performs the method of the request at 
state 455. The remote device's RPC manager then 
transmits a result aruJ returns to the waiting state 
451. 
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If the received packet contains a message that 
the remote device has already seen, the RPC man- 
ager of the intelligent remote device moves to state 
457 where the intelligent remote device determines 
whether the method invocation request was already s 
completed or whether it is still working on the 
request, tf the intelligent remote device has already 
completed the request, the subsystem retransmits 
the results at state 459 and returns to the waiting 
state 451. Othenvise. if the intelligent remote de- io 
vice is still working on the method invocation, the 
remote device's RPC manager moves to state 461 
where the remote device's RPC manager requests 
more lime to complete the invoked method and 
then returns to the waiting state 451. ;s 

Rgure 21c illustrates the sequence of a re- 
mote method invocation where there are no com- 
munication problems. At step 501. a user interface 
object in the hand-held display device requests a 
remote method invocation. Therefore, the user in- 20 
terface object sends a rnessage with a method 
selector and argument to the display device's RPC 
manager. The display device's RPC manager 
packs the message received into a packet at step 
502 and transmits the packet across the commu- 25 
rucations network at step 503. 

The intelligent remote device's RPC manager 
receives the transmitted packet at step 504. The 
remote device's RPC manager unpacks the mes- * 
sage from the packet at step 505 and directs the 30 
packet to the device driver program object within 
the intelligent remote devk:e. The device driver 
program object invokes the ,method specified by 
the method selector in the message at step 506. 
The device driver program object generates a re- 35 
turn message that is passed back to the remote 
device's RPC manager. At step 507. the remote 
device's RPC manager packs the return message 
into a reply buffer. The remote device's RPC man- 
ager sends the reply buffer at step 508. 40 

The display device's RPC manager receives 
the reply buffer at step 509 and unpacks the reply 
buffer to extract a return value at step 510. The 
display device's RPC manager delivers the return 
value to the requesting user interface program ob- 45 
ject that invoked the method in the device driver 
program object. Finally^ the user interface program 
object analyzes the return value at step 511. 

As illustrated, the present inventk)n can control 
an intelligent remote device by remotely invoking so 
methods in a device driver program object located 
within the intelligent remote device. In addition, it 
will be appreciated by the reader that the remote 
device may also remotely invoke methods in the 
display device. For example, the devce driver ob- 55 
ject in a remote device may invoke a method in a 
user interface object in the display device to dis- 
play a message. 



Many remote devices are not sophisticated 
enough to have internal control methods that can 
be invoked by the display device. Therefore, the 
present invention provides other methods lor con- 
trolling simple devices. 

Simple Device Control 

In the present invention, a remote devtee is 
defined as a "simple device* if the remote device 
does not have a microprocessor or the remote 
device contains such a primitive processor that the 
display device can not invoke control methods 
within the remote device. Figure Id illustrates, in 
block diagram form, the components in a simple 
remote device. To control a simple device, the 
hand-held display device of the present invention 
transmits simple predefined control codes instead 
of invoking control methods within the simple re- 
mote device- Therefore, to control a simple remote 
device, the hand-held display device must obtain 
both a user interface program object and a device 
driver program object from the simple remote de- 
vice. 

An example of obtaining user interface pro- 
gram object and a device driver program object 
from a simple remote device is given with refer- 
ence to Figures 22a - 22d. 23a - 23d and the 
ftowchart in Figure 24. Figures 22a - 22d illustrate 
on a high level how the remote simple device 155 
and the hand-held display device 170 of the 
present invention communicate with each other. 
Figures 23a - 23d illustrate, in conceptual form, 
how the user interface program object 353 and the 
device driver program object 351 of the remote 
external device 150 are transported the hand-held 
display device 170 and used. 

Referring to Figure 22a. a thermostat imple- 
mented as a simple remote device 155 is illus- 
trated. As specified by step 124 of Figure 24. the 
simple remote device 155 of Figure 22a transmits 
both a user interface program object and a device 
driver program object to the hand-heW display de- 
vice 170. Illustrated conceptually in Figure 23a. the 
simple device 155 transmits the user interface pro- 
gram object 353 and the device driver program 
object 351. The transmission of the program ob- 
jects is done using some communications means 
162 that may be a communications netvrark or any 
other means of transferring the information into the 
display device 170. 

The hand-held display device 170 receives the 
copy of the user interface program object 363 that 
descrit)es the simple device's graphical user inter- 
face. The hand-held display device 170 then in- 
vokes a method in the user interface program 
object 363 that renders a display object within a 
Space shown on the hand-heM display device 170. 
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When a user Selects the display object associated 
with the simple remote device 155, the display 
device 170 invokes a method within the user inter- 
face program object 363 as slated in step 125 of 
Figure 24 and illustrated in Figure 22b. The meth- 5 
od invoked in the user interface program object 
363 displays a graphical user interface associated 
with the simple remote device on the display 
screen 171 of the display device 170. The user can 
then interact with simple remote device's graphical 10 
user interface on display device 170 as stated in 
step 127. 

The user's interactions with the simple device's 
graphical user interface are analyzed by the simple 
device's device driver program object 361 within 75 
the display device 150. The device driver program 
object 361 resolves the user's interactions into con- 
trol codes that the simple remote device 155 will 
respond to. As illustrated in Rgure 23c, the device 
driver program object 361 located within the dis- 20 
play device 170 analyzes the user's actions. 

Referring to Figures 22c and 23c, the hand- 
held display device 170 transmits the control codes 
to the simple remote device 155. The simple re- 
mote device 155 changes in respi.^r^se to the con- 2s 
trol codes received as stated in step 130. Rnally. if 
the simple remote device 155 and the display 
device 170 are coupled together by a communica- 
tions network, the simple device 155 can transmit 
any return values or feedback generated by the 30 
simple remote device 155 as illustrated in Figure 
22d. 

Conventional Device Control 

as 

Existing remote devices that are not designed 
for the hand-held display device of the present 
invention do not have a user interlace program 
object to transmit to the hand-held display device. 
For example, Figure 1e illustrates a typical existing 40 
remote device that can only respond to coded 
command signals such as infrared signals. In such 
cases, the hand-heU display device must be man- 
ually informed about the remote device. The hand- 
held display device then uses a generic user inter- 45 
face program otjject for the type of remote device. 

For example, if a user has a conventional tele- 
vision that is controlled by infrared signals, the user 
enters a model numt)er of the television to inform 
\\\e hand-held display device afcxDut the convert- 50 
tiona! television. Tlie hand-held display device then 
accesses a database to obtain a list of the control 
codes Uiat the conventional television will under- 
stands. A television display object is then created 
to control the conventional television. If the user ss 
selects television display object, the hand-held dis- 
play device will invoke a method that displays a 
generic television user interface. When the user 



interacts with the generic television user interface, 
the display device will transniit the proper infrared 
control codes to control the television. 

Summary 

The foregoing has described an intuitive 
graphical user interface based upon a geographic 
map structure and a system for controlling remote 
external electronic devices. In the user interface of 
the present invention, each Space of the geo- 
graphic map structure is rendered on a touch 
screen display as a Space, Within each Space are 
colored icons called "objects' that can be Selected 
and manipulated by the user. Certain objects, re- 
ferred to as portals, transport the user from one 
Space to another Space when Selected. Other ob- 
jects, referred to as buttons, perform associated 
actions or functions when Selected. The user inter- 
face is displayed on a hand-held display device 
that can control remote electronic devices. Each 
remote electronic device transmits a user interface 
program object that defines a graphical user inter- 
face to the display device. The hand-held display 
device displays the graphical user interface asso- 
ciated with the remote device when a user selects 
an icon associated with the remote device on the 
display device's touch screen display. It is con- 
templated that changes and modifications may be 
made by one of ordinary skill in the art. to the 
materials and arrangements of elements of the 
present invention without departing from the scope 
of the invention. 

Claims 

1. A method of providing a graphical user inter- 
face on a computer system, said computer 
system comprising a CPU. a memory, and 
graphics display screen, said method compris- 
ing the steps of: 

providing a plurality of Spaces, each of 
said Spaces comprising a graphic background 
image: 

organizing said Spaces in a directed graph 
structure, said Spaces cormected together with 
a plurality of portals; 

locating said plurality of portals in said 
Spaces, each of said portals tiaving an asso- 
ciated target Space such that said portal con- 
nects the Space said portal is in with said 
associated target Space, each of said portals 
comprising a graphic icon: 

rendering a graphic background image of 
a first Space and portals located in said first 
Space on said graphics display screen, said 
first Space having at least a first portal: 

receiving input from a user, said input 
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comprising a portal selected from said portaJs 
located in said first Space; and 

rendering a graphic background image of 
a target Space associated with said selected 
portal on said graphics display screen; 5 
such that said user can move t^etween Spaces 
in said directed graph structure by selecting 
said portals within said Spaces. 

2. The method of providing a graphical user inter- 10 
face on a computer system as claimed in 
claim 1 further wherein said steps of rendering 
a background image further comprises the 
steps of: 

rendering a first subset of said background /5 
image on said graphics display screen, said 
first subset from a first location in said back- 
ground image; 

receiving directional input from said user, 
said directional input comprising a movement 20 
direction; and 

rendering a second subset of said back- 
ground image on said graphics display screen, 
said second subset from a second tocation in 
said background image, said second location 25 
being said first location plus an offset cor- 
responding to said movement direction; 
such that said user can move about within a 
background image associated with a Space. 

30 

3- The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 2 further wherein said step of receiving 
directional input from said user further com- 
prises detecting a finger of said user sliding 35 
across said display screen. 

4. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 1 further comprising the steps of: 40 

providing a Wayback portal, said Waytsack 
portal displayed in each of said Spaces, said 
Wayk>ack portal comprising a stack of data 
structures, each of said data structures com- 
prising a pointer, each of said pointers pointing 45 
to one of said Spaces; 

pushing a first data structure on said 
Wayback portal stack when said user moves to 
said first Space, said first data structure con- 
taining a pointer to said first Space; • 50 

popping a second data structure off of said 
Wayback portal stack when said user selects 
said Wayback portal, said second data struc- 
ture containing a pointer to a second Space; 
^ '55 

moving said user to sakJ second Space 
pointed to by said pointer in said second data 
structure popped off said Wayback portal 



Stack; 

such that said user can trace, back atong a 
path of Spaces whk:h said user travels 
through. 

5. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 4 wherein said data structures in said 
Wayback portal further comprises a coordi- 
nate, said coordinate identifying a location in 
said first Space where said user was when 
said user left said first Space. 

6. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 4 wherein said Wayback portal is ren- 
dered as a snapshot of a Space pointed to by 
a pointer in a data structure on lop of said 
Wayback portal stack. 

7. The method of providing a graphical user Inter- 
face on a computer system as claimed in 
claim t further comprising the steps of: 

providing a plurality of button objects, 
each of said button objects associated with a 
function; 

rendering said button objects within said 
Spaces, said button objects comprising graph- 
ic icons; and 

executing the function associated with a 
button object when said button object is se- 
tected by a user. 

8. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 1 wherein said display screen comprises 
a touch sensitive display screen. 

9. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim 8 wherein said input from said user 
comprises a finger touching said touch sen- 
sitive display screen. 

10. The method of providing a graphical user inter- 
face on a computer system as claimed in 
claim. 1. sakl method further comprising the 
steps of: 

rendering an anthropomorphic graphk: 
character on said display; 

attracting said user's attention by animat- 
ing said anthropomorphic graphic character; 
and 

conveying state information to said user by 
changir^ a visual appearance of said anth- 
ropomorphk: character. 



23 



45 



EP 0 626 635 A2 



46 



11. A method of displaying a plurality of objects 
for a user to select from on a display screen 
associated with a computer system, said meth- 
od comprising the steps of: 

organizing a plurality of objects into a cir- s 
cular object list; 

rendering a first subset of said objects in 
siaid circular object Hst as a first subset of 
graphic objects on said display screen, said 
graphic objects rendered on said display ;o 
screen aligned along a first axis; and 

rotating through said circular object list 
when said user slides a pointing device across 
said graphic objects in a direction substantially 
parallel to said ftrst axis such that said first i5 
subset of graphic objects on said display 
screen is replaced by a second subset graphic 
objects on said display screen, said second 
subset graphic objects on said display screen 
corresponding to a second subset of said ob- 20 
jects in said circular object list; 
such that said user can choose any of said 
plurality of objects by rotating through the cir- 
cular object list until a desired object appears 
on said display screen. 25 

12. The method of displaying a plurality of graphic 
objects for a user to select from as claimed in 
claim 11 further comprising. the step of: 

selecting a graphic otjject displayed on 30 
saiid display screen when a user places a 
pointing device on said graphic object dis- 
played on sard display screen for a minimum 
amount of time. 

35 

13. The method of displaying a plurality of graphic 
objects for a user to select from as claimed in 
claim 1 1 further comprising the step of: 

moving a graphic object off said circular 
object list when said user places said pointir^ 40 
device on said graphic object and nu>ves said 
pointing device in a direction substantially per- 
pendicular to said first axis. 

14. The method of displaying a plurality of graphic 45 
objects for a user to select from as claimed in 
claim 13 wherein said display screen com- 
prises a touch screen and said pointing device 
comprises a finger. 

50 

15. The method of displaying a plurality of graphic 
objects for a user to select from as claimed in 
claim 1 1 wfterein said step of rotating through 
said circular object list is enhanced by simulat- 
ing a physical wheel that is affected by friction. ss 

16w In a graphical user interface for a computer 
system, said graphical user interface compris- 



ing a plurality of Spaces organized in a di- 
rected graph structure, each of said Spaces 
comprising a graphic background image, said 
Spaces coupled in said directed graph struc- 
ture using a plurality of portals, each of said 
portals providing a connection between a first 
Space and a second Space, a method for 
storing a path traveled along said directed 
graph structure, said method comprising the 
steps of: 

rendering a graphic background image 
correspoTKling to a first Space on a display 
screen, said first Space including at least one. 
portal, said portal associated with a target 
Space; . 

providing a Wayback portal, said Wayback 
portal displayed in each of said Spaces, said 
Wayback portal comprising a stack of Space 
pointers, each of said Space pointers pointing 
to a previously visited Space; 

receiving input from a user, said input 
comprising said Wayback portal or a portal 
selected from said portals k)cated in said first 
Space; and 

processing said input received from said 
user such that 

if said user selects a portal, said user 
interlace pushing a pointer to a Space asso- 
ciated with said selected portal on said 
Wayback stack and rendering a graphic back- 
ground image of the Space associated with 
said selected portal on said graphics display 
screen, else 

if said user selects said Wayt>ack portal, 
said user interface popping a Space pointer off 
the top of said Wayt>ack portal stack and ren- 
dering said Space indicated by said Space 
pointer on said graphics display screen. 

17. The method of storing a path traveled ak)ng 
said directed graph structure in a graphical 
user interface for a computer system as 
claimed in claim 11 wtierein said Wayback 
portal comprises a mirvature image of the 
Space indicated by the pointer on the top of 
the Wayback portal stack. 

18. In a computer system fiavir>g a graphical user 
interface, an animated multipurpose icon, said 
icon comprising the elements of: 

an anthropomorphic graphic character ren- 
dered on said display; 

means for attracting a user's attention by 
animating said anthropomorphic graphic char- 
acter, and 

means lor conveying state information by 
changing a visual appearance of said anth- 
ropomorphic character. 
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19. In a graphical user tnterface for a computer 
system, said graphical user interface compris- 
ing a plurality of graphic objects rerKtered on a 
display screen, said display screen comprising 
a touch screen which senses a finger placed s 
on the screen and determines a location of 
said finger on said touch screen, a method of 
allowing a user to grab a graphic object said 
method comprising the steps of: 

rendering a graphic object on said touch io 
screen display, said graphic object defining an 
area on said touch screen display; 

placing a finger on said touch screen dis- 
play near said graphic object rendered on said 
touch screen display; ,5 

circling said graphic object rendered on 
said touch screen display with said finger; 

touching said graphic object rendered on 
said touch screen display; and 

rendering said graphic object on said 20 
touch screen display at a location defined by 
said finger on said display screen as deter- 
mined by said touch screen display; 
such that a user can select said graphic object 
by circling said graphic object with a fmger 25 

20. An apparatus for controlling at least one re- 
mote device, said apparatus comprising the 
elements of: 

a central processing unit (CPU); 30 

a memory, said memory coupled to said 
central processing unit; 

a display screen, .said display screen 
coupled to said CPU. said display screen dis- 
playing graphic images: 35 

communications means for receiving and 
transmitting messages across a communica- 
tions network, said communications means 
coupled to said central processing unit; 

means for receiving a user interface pro- 4o 
gram object over said communications means, 
said user interface program object broadcast 
by a remote device, said user interface pro- 
gram object defining a user interface for said 
remote device: 45 

means for invoking a user interface meth- 
od in said user interface program object, said 
user interface method displaying a graphical 
user interface associated with said remote de- 
vice on said cfisplay screen; 50 

means for accepting input from a user, 
said input guided by said graphk:al user inter- 
face associated with said remote device on 
said display saeen; and 

means for invoking a device driver method ss 
associated with said remote device in re- 
sponse to said input from said user, said de- 
vice driver method controlling said remote de- 



vice; 

such that said user interface of said remote 
device is displayed on said display screen and 
interaction with said user interface invokes a 
device driver method. 

21. The apparatus for controlling at least one re- 
mote device as claimed in claim 20. said ap- 
paratus further comprising 

means for receiving a device driver pro- 
gram object associated with said remote de- 
vice over said communications means, said 
device driver, program object including said 
device driver method for controlling said re- 
mote device. 

22. The apparatus for controlling at least one re- 
mote device as claimed in claim 21. said ap- 
paratus further wherein said device driver 
method is broadcast by said remote device 
across said communications network and 
copied into said apparatus. 

23. The apparatus for controlling at least one re- 
mote device as claimed in claim 22 wherein 
said device driver method translates said input 
from said user into a plurality of control codes. 

24. The apparatus for controlling at least one re- 
mote device as claimed in claim 23 wherein 
said apparatus transmits said plurality of con- 
trot codes to said remote device. 

25. The apparatus for controlling at least one re- 
mote device as claimed in claim 20 wherein 
said device driver method is located within 
said remote device. 

26. The apparatus for controlling at least one re- 
mote device as claimed in claim 25 wherein 
said device driver method is invoked remotely 
across said communications network by said 
apparatus. 

27. The apparatus for controlling at least one re- 
mote device as claimed in claim 26 wherein 
said device driver method is refrK>tely invoked 
using a robust remote procedure call manager. 

28. A method of obtaining a graphical user inter- 
face from said remote device and controlling 
said renu>te device with a kxal display device, 
said local display device coupled to a commu- 
nication network, said method comprising: 

connecting said remote device to said 
communication network; 

signaling from said remote device that said 
remote device has a graphical user interface to 
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interface displayed on said local display de- 
vice: and 

invoking a control method in a device driv- 
er program object to handle said interactions 
with said remote device's user interface. 25 

30. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 29. wherein 30 
said device driver program object is located 
within said remote device. 

31. The method of obtaining a remote device's 
user interface from said remote device and 35 
controlling said remote device with a local dis- 
play device as claimed in claim 3D, wherein 
said control method in said device driver pro- 
gram object is invoked remotely across said 
communications network. 40 

32. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 29. said meth- 45 
od further comprising the step of transmitting 

said device driver program object across said 
communications network to said k)cai display 
device. 

50 

33- The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a tocal dis- 
play device as claimed in claim 32, wherein 
said device driver method within said device 55 
driver program object is invoked within said 
k)cal display device. 



34. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis- 
play device as claimed in claim 33. wherein 
said device driver method translates said user 
input into commands codes. 

35. The method of obtaining a remote device's 
user interface from said remote device and 
controlling said remote device with a local dis* 
play device as claimed in claim 34, wherein 
said local display device transmits said com- 
mands codes to said remote device. 

3te. A method of copying an object in a graphical 
user interface, said method comprising the 
steps of: 

rendering a first graphic image of said 
object at a first position on a graphics display 
screen; 

receiving input from a user, said input 
comprising a command to copy said object; 

rendering a ghost image of said object at 
said first position on said display screen, said 
ghost image comprising a copy of said first 
graphic image of said object drawn in muted 
colors; 

receiving input from a user, said input 
comprising a second position to locate a copy 
of said object; 

rendering said first graphic image of said 
object at said second position; and 

changing the ghost image of said object at 
said first position on said display screen such 
that said ghost image of said object becomes 
identical to said first graphic image of said 
object; 

such that two graphic images of said object 
now exist. 

37. The method of copying an object in a graphi- 
cal user interface as claimed in claim 36. 
wherein said input comprises a finger touching 
a touch screen display 

38. A method of nooving an object in a graphical 
user interface, said method comprising the 
steps of: 

rendering a first graphic image of said 
object at a first position on a graphics display 
screen; 

receiving input from a user, said input 
comprising a command to move said object; 

rendering a ghost Image of said object at 
said first position on said display screen, said 
ghost image comprisirtg a copy of said first 
graphic image of said object drawn in muted 
cotors; 



export upon t»eing connected to said commu- 
nication r^etwork; 

requesfing said graphical user interface by 
said kjcal display device; 

transmitting a graphical user interlace pro- 5 
gram object to said local display device, said 
graphical user interface program object con- 
taining a method which display said graphical 
user interface; and 

invoking said method in said user interface 10 
program object such that said remote device's 
graphical user interface is displayed on said 
local display device. 

The method of obtaining a remote device's is 
user interface from said remote device and 
controlling said remote device with a bcal dis- 
play device as claimed in claim 28. wherein 
said method further comprises the steps of: 

interacting with said remote device's user 20 
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user currently is within said map structure; 

displaying a plurality of portals in said first 
Space, each of said portals having an asso- 
ciated target Space such that said portal con- 
nects the Space said portal is in with said 
associated target Space such that each portal 
provides said user a place to go. each of said 
portals comprising a graphic icon: and 

displaying a Wayback portal in said first 
Space, said Wayback depicting a image of a 
Space said user just left. 



receiving input from a user, said input 
comprising a second position to locate a said 
object; 

rendering said second graphic image of 
said object at said second position; and 

moving said ghost image of said object at 
said first position to said second position such 
that sakJ ghost image of said object merges 
into said second graphic Image of said object 
at said second position; 

such that one graphic image of said object 
now exists at said second position. 

39. The method of copying an object in a grapN- 
cal user interface as clamed in claim 38, is 
wherein said input comprises a finger touching 

a touch scr&en display 

40. In an object-oriented graphical user interface, 
said object-oriented graphic user interface il- 
lustrating objects as graphical icons, a method 
of providing information to a program object, 
said method comprisirig the steps of: 

rendering a graphic image of a data object 
at a first position on a graphics display screen; 

receiving input from a user, said input 
comprising a command to move said data ob- 
ject; 

rendering a ghost image of said object at 
said first position on said display screen, said 
ghost image comprising a copy of said first 
graphic image of said object drawn in muted 
colors; 

receiving input from a user, said input 
comprising a second position to k)cate a said 
data object, said program object located at 
said second position: and 

receiving input from a user, said input a 
command to provide sakl data object to said 
program object; 

such that two grapliic images of said object 
now exist 



4a The method of navigating through a graphical 
user interface as claimed in claim 42. wherein 
said map structure is organized as a directed 
graph. 



44. The method of navigating through a graphical 
user interface as claimed in claim 43, wherein 
if a user selects one of sard portals then said 
user is moved to a Space associated with said 
Portal. 



41. The method of providing information to a pro- 
gram object as claimed in claim 40, wherein 45 
said input comprises a finger touching a touch 
screen display 



4Z A method of navigating through a graphical 

user interface, said graphiical user interface so 
displayed on a display screen, said graphical 
user interface comprising a plurafity of Spaces, 
each of said Spaces comprising a graphic 
background image, said Spaces organized into 
a map structure, said method comprising the 55 
steps of: 

displaying a first Space on said display 
screen, said first Space representing virhere a 



27 



EP 0 626 635 A2 




EP 0 626 635 A2 



170 



MEMORY 



•33 



CPU 



A 



•30 



INPUT/OUTPUT 



^34 



EXTERNAL 
COMMUNICATION 



VCR 





39 



^42 



OTHER INPUT 
DEVICES 

(MOUSE, 
TRACKBALL, 
JOYSTICK) 



' J 



^37 



TOUCH 
SCREEN 
DISPLAY 



THERMOSTAT 




150 



Figure lb 



EP0 626 635 A2 



401 



403 



404 



Communication 
System 



CPU 



Memory 



303 



Device Control 



405 



Figure Ic 



-150 



420 



424 



Emitter 



Receiver 



Memory 



353 



Device Control 

Circuitry 



421 425 

Figure Id 



-155 



450 



455 



Receiver 



Device Control 
Circuitry 



-157 



Figure le 

(Prior Art) 



EP 0 626 635 A2 




YOUR TV YOUR VCR 

Figure 2a 



EP 0 626 635 A2 




EP 0 626 635 A2 




c 



EP 0 626 635 A2 





Murder,S he Spoke 

WKilejessicof Angela Lansbury)is recording one 
of her novel* for the Mystery Books for the Blind 
Series, a murder occurs With Michael Cole, 
Charlie Daniels andjonnalee. 
Or 



our. 





EP 0 626 635 A2 




Figure 11 



CO 

CO 



EP0 626 635 A2 




EP 0 626 635 A2 




EP 0 626 635 A2 





C4 




0 626 635 A2 



Figure 17a 



Figure 17b 



Figure 17c 



BP 0 626 635 A2 

THERMOSTAT 




Figure 18a 




Figure 18c 



THERMOSTAT 




Figure 18d 



EP 0 626 635 A2 



150— 
301 



--A 



150— 
301 



-A 



150— 
301 



150— 
301 



.303 



\ 

160 



— 170 



307 



309 



Figure 19a 



.303 



313- 



\ 

160 



-170 
-311 



307 309 

Figure 19b 




,303 



313- 



\ 

160 



— 170 
-311 



307 



309 



Figure 19c 




.303 



313- 



N 

160 



f 



— 170 
-311 



307 309 

Figure 19d 



EP 0 626 635 A2 



REMOTE DEVICE IS TURNED ON 

AND BEGINS ANNOUNCING 
IT HAS AN INTERFACE TO EXPORT 



— 101 



I 



DISPLAY DEVICE RECEIVES EXPORT OP 
INTERFACE NOTICE FROM THE DEVICE 
AND SIGNALS BACK TO BEGIN TRANSMITTING 



— 102 



I 



REMOTE DEVICE BROADCASTS PROGRAM 
OBJECTS FOR ITS INTERFACE 



— 103 



DISPLAY DEVICE RECEIVES PROGRAM OBJECTS 
FROM REMOTE DEVICE AND DISPLAYS OBJECT 
ON THE DISPLAY SCREEN 



— 104 



I 



DISPLAY DEVICE INVOKES REMOTE DEVICE S 
USER INTERFACE PROGRAM OBJECT WHEN 
REMOTE DEVICE DISPLAY OBJECT SELECTED 



— 105 



I 



USER INTERACTS WITH REMOTE 
DEVICE'S GUI ON DISPLAY DEVICE 



— 106 



CONTROL METHOD INVOKED ON CORRECT 
REMOTE DEVICE AS INDICATED BY REMOTE 
NAME SERVER, TO CONTROL REMOTE DEVICE 



— 107 



REMOTE DEVICE CHANGES IN RESPONSE TO 
CONTROL METHOD INVOCATION 



— 108 



I 



RETURN VALUES, IF ANY, ARE BROADCAST BACK 
TO THE DISPLAY DEVICE 



— 109 



Figure 20 



EP 0 626 635 A2 



FINITE STATE MACHINE 
FOR THE REQUESTING SIDE OF A 
REMOTE METHOD INVOCATION 




RETRIES > = 
, LIMIT 




Figure 21a 



EP 0 626 635 A2 



FINITE STATE MACHINE 
FOR THE RECEIVING SIDE 
OF A REMOTE METHOD 




EP 0 626 635 A2 




EP 0 626 635 A2 



155- 







424 

/ 








Ml II IIIIB 


162 

\ 














420 \ 








1 EMITTER 1 




1 


) ^^-^ RECEIVER 1 





170 



>171-. 



Figure 22a 

•170 



Figure 22b 




155- 



THERMOSTAT 



424 

(memory) 



1 



420 \ 

I emitter] 



< |receiver} <- 



421 



170 




Figure 22c 



thermostat 



155-* — 




Figure 22d 



EP 0 626 635 A2 




Figure23a 

3S3 



156— 




Figure 23b 

353 




421 363 



Figure 23d 



EP 0 626 635 A2 



REMOTE DEVICE S USER INTERFACE AND CONTROL 
PROGRAM CODE TRANSFERRED TO 
DISPLAY DEVICE 


—124 


r- i 


—125 


DISPLAY DEVICE INVOKES USER 
INTERFACE PROGRAM CODE 








DISPLAY DEVICE DISPLAYS REMOTE DEVICE S 
GRAPHICAL USER INTERFACE 


—126 
—127 


A 


USER INTERACTS WITH DEVICE'S GUI ON DISPLAY 
DEVICE, WHICH CAUSES A CONTROL METHOD 
TO BE INVOKED IN THE DISPLAY DEVICE 


\ 


r 




METHOD INVOCATION RESOLVES THE USER S 
ACTIONS INTO A CONTROL CODE FOR 
THE REMOTE DEVICE 


—128 
—129 
—130 




f 


CONTROL CODE IS TRANSMITTED TO REMOTE 
DEVICE VIA SOME COMMUNICATION MEANS 




r 


REMOTE DEVICE CHANGES IN RESPONSE TO 
THE CONTROL CODE EXECUTION 




r 




RETURN VALUES, IF ANY, ARE BROADCAST BACK 
TO THE DISPLAY DEVICE 


-131 



Figure 24 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFER£NCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



